79 lines
2.9 KiB
Markdown
79 lines
2.9 KiB
Markdown
|
# Project Hints
|
||
|
|
|
||
|
|
## Overview
|
||
|
|
This is an unofficial TypeScript client for accessing abuse.ch threat intelligence APIs, including URLhaus, ThreatFox, and FeodoTracker.
|
||
|
|
|
||
|
|
## Recent Changes (2025-11-22)
|
||
|
|
|
||
|
|
### Dependency Upgrades
|
||
|
|
- **Migrated from @gitzone to @git.zone scope packages**
|
||
|
|
- All dev dependencies now use @git.zone/* scope (the correct new scope)
|
||
|
|
- @git.zone/tsbuild: 3.1.0
|
||
|
|
- @git.zone/tsbundle: 2.5.2
|
||
|
|
- @git.zone/tsdoc: 1.9.2
|
||
|
|
- @git.zone/tsrun: 2.0.0
|
||
|
|
- @git.zone/tstest: 3.1.3
|
||
|
|
|
||
|
|
- **Removed @push.rocks/tapbundle**
|
||
|
|
- Now using `import { tap, expect } from '@git.zone/tstest/tapbundle'`
|
||
|
|
- Tests updated to use `export default tap.start()` pattern
|
||
|
|
|
||
|
|
- **Updated production dependencies**
|
||
|
|
- @push.rocks/smartfile: 11.2.7 (stayed on v11 to avoid v13 breaking changes)
|
||
|
|
- @push.rocks/smartpath: 6.0.0
|
||
|
|
- csv-parser: 3.2.0
|
||
|
|
- https-proxy-agent: 7.0.6
|
||
|
|
- unzipper: 0.12.3
|
||
|
|
|
||
|
|
### TypeScript Configuration
|
||
|
|
- Fixed tsconfig.json to use `module: "nodenext"` to match `moduleResolution: "nodenext"`
|
||
|
|
- Build command works with `tsbuild --allowimplicitany` (without --web flag to avoid type errors in node_modules)
|
||
|
|
|
||
|
|
### File Naming
|
||
|
|
- Note: There was a typo in the file naming - originally had `urlhouse.ts`, now correctly named `urlhaus.ts`
|
||
|
|
|
||
|
|
## Architecture
|
||
|
|
|
||
|
|
### Main Classes
|
||
|
|
- **UrlHaus** - Fetches malicious URL data from URLhaus CSV export
|
||
|
|
- **ThreatFox** - Fetches IOC data from ThreatFox CSV export
|
||
|
|
- **FeodoTracker** - Fetches botnet C&C server data from FeodoTracker JSON API
|
||
|
|
|
||
|
|
### Data Flow
|
||
|
|
1. Classes fetch data from abuse.ch APIs
|
||
|
|
2. For CSV data (URLhaus, ThreatFox): Downloads, extracts from ZIP, parses CSV
|
||
|
|
3. For JSON data (FeodoTracker): Direct JSON fetch and parse
|
||
|
|
4. Returns typed TypeScript interfaces
|
||
|
|
|
||
|
|
### Dependencies
|
||
|
|
- **smartfile v11.2.7** - Used for:
|
||
|
|
- `plugins.smartfile.fs.ensureDirSync()` - Create temporary directories
|
||
|
|
- `plugins.smartfile.fs.removeSync()` - Clean up temporary files
|
||
|
|
- **smartpath v6.0.0** - Used for:
|
||
|
|
- `plugins.smartpath.get.dirnameFromImportMetaUrl()` - Get package directory from import.meta.url
|
||
|
|
- **node-fetch** - HTTP requests to abuse.ch APIs
|
||
|
|
- **csv-parser** - Parse CSV data from URLhaus and ThreatFox
|
||
|
|
- **unzipper** - Extract downloaded ZIP files
|
||
|
|
- **https-proxy-agent** - Proxy support for HTTP requests
|
||
|
|
|
||
|
|
## Testing
|
||
|
|
- Tests use @git.zone/tstest framework
|
||
|
|
- Run with: `pnpm test`
|
||
|
|
- Tests verify data retrieval from all three abuse.ch services
|
||
|
|
- All tests currently passing ✅
|
||
|
|
|
||
|
|
## Build
|
||
|
|
- Build with: `pnpm build`
|
||
|
|
- Uses @git.zone/tsbuild
|
||
|
|
- Outputs to: `dist_ts/`
|
||
|
|
- Note: Use without --web flag to avoid type errors in node_modules
|
||
|
|
|
||
|
|
## Known Issues
|
||
|
|
- None currently
|
||
|
|
|
||
|
|
## Important Notes
|
||
|
|
- This module accesses free community threat intelligence data
|
||
|
|
- Be respectful of abuse.ch resources - avoid excessive polling
|
||
|
|
- The module automatically detects and uses HTTP_PROXY/HTTPS_PROXY environment variables
|
||
|
|
- All classes return Promises - use async/await
|