fix(smarts3): replace TypeScript server with Rust-powered core and IPC bridge
This commit is contained in:
102
readme.hints.md
102
readme.hints.md
@@ -1,74 +1,60 @@
|
||||
# Project Hints for smarts3
|
||||
|
||||
## Current State (v3.0.0)
|
||||
## Current State (v6.0.0-dev)
|
||||
|
||||
- Native custom S3 server implementation (Smarts3Server)
|
||||
- No longer uses legacy s3rver backend (removed in v3.0.0)
|
||||
- Core S3 operations working: PUT, GET, HEAD, DELETE for objects and buckets
|
||||
- Multipart upload NOT yet implemented (critical gap for production)
|
||||
- Authentication is hardcoded ('S3RVER'/'S3RVER') - not production-ready
|
||||
- No CORS support yet
|
||||
- No SSL/TLS support yet
|
||||
- **Rust-powered S3 server** via `@push.rocks/smartrust` IPC bridge
|
||||
- High-performance: streaming I/O, zero-copy, backpressure, range seek
|
||||
- TypeScript is thin IPC wrapper; all HTTP/storage/routing in Rust binary `rusts3`
|
||||
- Full S3 compatibility: PUT, GET, HEAD, DELETE for objects and buckets
|
||||
- Multipart upload support (streaming, no OOM)
|
||||
- Authentication (AWS v2/v4 signature key extraction)
|
||||
- CORS support
|
||||
- ListBuckets, ListObjects (v1/v2), CopyObject
|
||||
|
||||
## Production Readiness
|
||||
## Architecture
|
||||
|
||||
See `production-readiness.md` for the complete gap analysis and implementation plan.
|
||||
### Rust Binary (`rust/src/`)
|
||||
- `main.rs` - Clap CLI, management mode entry
|
||||
- `config.rs` - Serde config structs matching TS interfaces
|
||||
- `management.rs` - IPC loop (newline-delimited JSON over stdin/stdout)
|
||||
- `server.rs` - hyper 1.x HTTP server, routing, CORS, auth, all S3 handlers
|
||||
- `storage.rs` - FileStore: filesystem-backed storage, multipart manager
|
||||
- `xml_response.rs` - S3 XML response builders
|
||||
- `s3_error.rs` - S3 error codes with HTTP status mapping
|
||||
|
||||
**Key Missing Features for Production:**
|
||||
1. Multipart upload support (HIGHEST PRIORITY)
|
||||
2. Configurable authentication
|
||||
3. CORS middleware
|
||||
4. SSL/TLS support
|
||||
5. Production configuration system
|
||||
6. Production logging
|
||||
### TypeScript Bridge (`ts/`)
|
||||
- `ts/index.ts` - Smarts3 class with RustBridge<TRustS3Commands>
|
||||
- `ts/plugins.ts` - path, smartpath, RustBridge, tsclass
|
||||
- `ts/paths.ts` - packageDir, bucketsDir defaults
|
||||
|
||||
## Architecture Notes
|
||||
### IPC Commands
|
||||
| Command | Params | Action |
|
||||
|---------|--------|--------|
|
||||
| `start` | `{ config: ISmarts3Config }` | Init storage + HTTP server |
|
||||
| `stop` | `{}` | Graceful shutdown |
|
||||
| `createBucket` | `{ name: string }` | Create bucket directory |
|
||||
|
||||
### File Structure
|
||||
- `ts/classes/smarts3-server.ts` - Main server class
|
||||
- `ts/classes/filesystem-store.ts` - Storage layer (filesystem-backed)
|
||||
- `ts/classes/router.ts` - URL routing with pattern matching
|
||||
- `ts/classes/middleware-stack.ts` - Middleware execution
|
||||
- `ts/classes/context.ts` - Request/response context
|
||||
- `ts/classes/s3-error.ts` - S3-compatible error handling
|
||||
- `ts/controllers/` - Service, bucket, and object controllers
|
||||
- `ts/index.ts` - Main export (Smarts3 class)
|
||||
### Storage Layout (backward-compatible)
|
||||
- Objects: `{root}/{bucket}/{key}._S3_object`
|
||||
- Metadata: `{root}/{bucket}/{key}._S3_object.metadata.json`
|
||||
- MD5: `{root}/{bucket}/{key}._S3_object.md5`
|
||||
- Multipart: `{root}/.multipart/{upload_id}/part-{N}`
|
||||
|
||||
### Storage Layout
|
||||
- Objects stored as: `{bucket}/{encodedKey}._S3_object`
|
||||
- Metadata stored as: `{bucket}/{encodedKey}._S3_object.metadata.json`
|
||||
- MD5 stored as: `{bucket}/{encodedKey}._S3_object.md5`
|
||||
- Keys are encoded for Windows compatibility (hex encoding for invalid chars)
|
||||
## Build
|
||||
|
||||
### Current Limitations
|
||||
- Max file size limited by available memory (no streaming multipart)
|
||||
- Single server instance only (no clustering)
|
||||
- No versioning support
|
||||
- No access control beyond basic auth
|
||||
|
||||
## Testing
|
||||
|
||||
- Main test: `test/test.aws-sdk.node.ts` - Tests AWS SDK v3 compatibility
|
||||
- Run with: `pnpm test`
|
||||
- Tests run with cleanSlate mode enabled
|
||||
- `pnpm build` runs `tsrust && tsbuild --web --allowimplicitany`
|
||||
- `tsrust` compiles Rust to `dist_rust/rusts3`
|
||||
- Targets: linux_amd64, linux_arm64 (configured in npmextra.json)
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `@push.rocks/smartbucket` - S3 abstraction layer
|
||||
- `@push.rocks/smartfs` - Modern filesystem operations with Web Streams API (replaced smartfile)
|
||||
- `@push.rocks/smartxml` - XML generation/parsing
|
||||
- `@push.rocks/smartrust` - RustBridge IPC bridge
|
||||
- `@push.rocks/smartpath` - Path utilities
|
||||
- `@tsclass/tsclass` - TypeScript utilities
|
||||
- `@tsclass/tsclass` - IS3Descriptor type
|
||||
- `@git.zone/tsrust` (devDep) - Rust cross-compilation
|
||||
|
||||
## Migration Notes (2025-11-23)
|
||||
## Testing
|
||||
|
||||
Successfully migrated from `@push.rocks/smartfile` + native `fs` to `@push.rocks/smartfs`:
|
||||
- All file/directory operations now use smartfs fluent API
|
||||
- Web Streams → Node.js Streams conversion for HTTP compatibility
|
||||
- All tests passing ✅
|
||||
- Build successful ✅
|
||||
|
||||
## Next Steps
|
||||
|
||||
Waiting for approval to proceed with production-readiness implementation.
|
||||
Priority 1 is implementing multipart uploads.
|
||||
- `test/test.aws-sdk.node.ts` - AWS SDK v3 compatibility (10 tests)
|
||||
- `test/test.ts` - SmartBucket integration (3 tests)
|
||||
- Run: `pnpm test` or `tstest test/test.aws-sdk.node.ts --verbose`
|
||||
|
||||
Reference in New Issue
Block a user