golib/archive/doc.go

/*
 * MIT License
 *
 * Copyright (c) 2025 Nicolas JUHEL
 *
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in all
 * copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 * SOFTWARE.
 *
 */

// Package archive provides high-performance archive and compression management for Go
// applications with streaming-first operations, memory efficiency, and thread-safe primitives.
//
// # Overview
//
// This package offers production-ready archive and compression capabilities through three
// main subpackages, each addressing specific concerns while working together seamlessly:
//
//   - archive: Multi-file container management (TAR, ZIP)
//   - compress: Single-file compression algorithms (GZIP, BZIP2, LZ4, XZ)
//   - helper: High-level compression/decompression pipelines with thread-safe buffering
//
// The root package provides convenience wrappers for format detection and extraction,
// making it simple to handle archives and compressed files without knowing their format
// in advance.
//
// # Design Philosophy
//
//  1. Stream-First: All operations use io.Reader/io.Writer for continuous data flow
//  2. Memory Efficient: Constant memory usage regardless of file size (O(1))
//  3. Thread-Safe: Proper synchronization primitives for concurrent operations
//  4. Format Agnostic: Automatic detection via magic number analysis
//  5. Composable: Independent subpackages that integrate seamlessly
//
// # Key Features
//
// Stream Processing:
//   - Handle files of any size with constant memory (~10MB for 10GB archives)
//   - Zero-copy operations where possible
//   - Configurable buffer sizes for performance tuning
//
// Thread Safety:
//   - Atomic operations (atomic.Bool) for state management
//   - Mutex protection (sync.Mutex) for shared buffer access
//   - Goroutine synchronization (sync.WaitGroup) for lifecycle
//   - Safe for concurrent use across multiple goroutines
//
// Format Support:
//   - Archives: TAR (streaming), ZIP (random access)
//   - Compression: GZIP, BZIP2, LZ4, XZ, uncompressed
//   - Auto-detection via header analysis
//
// Security:
//   - Path sanitization against directory traversal
//   - Metadata preservation (permissions, timestamps, symlinks)
//
// # Architecture
//
// The package follows a layered architecture where each subpackage has specific
// responsibilities:
//
//	┌─────────────────────────────────────────────────────────┐
//	│                    Root Package                         │
//	│  ExtractAll(), DetectArchive(), DetectCompression()     │
//	└──────────────┬─────────────┬─────────────┬──────────────┘
//	               │             │             │
//	      ┌────────▼─────┐  ┌────▼─────┐  ┌────▼────────┐
//	      │   archive    │  │ compress │  │   helper    │
//	      │              │  │          │  │             │
//	      │ TAR, ZIP     │  │ GZIP, XZ │  │ Pipelines   │
//	      │ Reader/Writer│  │ BZIP2,LZ4│  │ Thread-safe │
//	      └──────────────┘  └──────────┘  └─────────────┘
//
// Component Characteristics:
//
//	archive:  O(1) TAR, O(n) ZIP memory; Thread-safe
//	compress: O(1) memory; Thread-safe; Stream-based
//	helper:   O(1) memory; Thread-safe; Async I/O
//	Root:     O(1) memory; Delegates to subpackages
//
// # Basic Usage
//
// Extract Archive (Auto-Detection):
//
//	file, err := os.Open("archive.tar.gz")
//	if err != nil {
//	    return err
//	}
//	defer file.Close()
//
//	// Automatically detects format and extracts
//	err = archive.ExtractAll(file, "archive.tar.gz", "/output")
//	if err != nil {
//	    return err
//	}
//
// Detect Compression Format:
//
//	file, err := os.Open("file.gz")
//	if err != nil {
//	    return err
//	}
//	defer file.Close()
//
//	// Detect compression and get decompressor
//	alg, reader, err := archive.DetectCompression(file)
//	if err != nil {
//	    return err
//	}
//	defer reader.Close()
//
//	// Use the detected algorithm
//	fmt.Printf("Detected: %s\n", alg.String())
//
// Parse Algorithm by Name:
//
//	// Parse compression algorithm from string
//	alg := archive.ParseCompression("gzip")
//	if alg.IsNone() {
//	    return errors.New("unknown compression")
//	}
//
//	// Parse archive algorithm from string
//	archAlg := archive.ParseArchive("tar")
//	if archAlg.IsNone() {
//	    return errors.New("unknown archive format")
//	}
//
// # Format Detection
//
// The package uses magic number detection for reliable format identification:
//
// Compression Magic Numbers:
//   - GZIP:  \x1f\x8b (1F 8B)
//   - BZIP2: BZ (42 5A)
//   - LZ4:   \x04\x22\x4d\x18 (04 22 4D 18)
//   - XZ:    \xfd7zXZ\x00 (FD 37 7A 58 5A 00)
//
// Archive Magic Numbers:
//   - TAR:   ustar at offset 257
//   - ZIP:   PK\x03\x04 (50 4B 03 04)
//
// Detection Process:
//  1. Read header bytes (minimum 6 bytes for compression, 265 for archives)
//  2. Compare against known magic numbers
//  3. Return algorithm and appropriate reader/writer
//  4. Preserve unread data for subsequent processing
//
// # Extraction Process
//
// ExtractAll handles nested compression automatically:
//
//  1. Detect compression layer (if present)
//  2. Decompress and recurse (handles .tar.gz, .tar.bz2, etc.)
//  3. Detect archive format
//  4. Extract files with metadata preservation
//  5. Sanitize paths against directory traversal
//
// Path Security:
//   - Removes "../" sequences to prevent traversal attacks
//   - Creates directories with appropriate permissions
//   - Preserves symlinks and hard links safely
//
// # Performance Characteristics
//
// Memory Efficiency:
//   - Constant memory usage regardless of file size
//   - Streaming architecture (512-byte chunks)
//   - Zero-copy for uncompressed data
//   - Example: 10GB archive uses ~10MB RAM
//
// Throughput Benchmarks (AMD64, Go 1.24, SSD):
//   - TAR Create: ~500 MB/s
//   - TAR Extract: ~400 MB/s
//   - ZIP Create: ~450 MB/s
//   - ZIP Extract: ~600 MB/s
//   - GZIP Compress: ~150 MB/s
//   - GZIP Decompress: ~300 MB/s
//   - BZIP2: ~20 MB/s (high compression)
//   - LZ4: ~800 MB/s (fastest)
//   - XZ: ~10 MB/s (best ratio)
//
// Algorithm Selection Guide:
//
//	Speed:       LZ4 > GZIP > BZIP2 > XZ
//	Compression: XZ > BZIP2 > GZIP > LZ4
//
//	Recommended:
//	├─ Real-time/Logs → LZ4
//	├─ Web/API → GZIP
//	├─ Archival → XZ or BZIP2
//	└─ Balanced → GZIP
//
// # Use Cases
//
// Backup Systems:
//   - Stream large directories to TAR.GZ without memory exhaustion
//   - Incremental backups with selective extraction
//   - Parallel compression across multiple jobs
//
// Log Management:
//   - Real-time compression of rotated logs (LZ4 for speed)
//   - Extract specific log files without full decompression
//   - High-volume logging with minimal CPU overhead
//
// CI/CD Pipelines:
//   - Package build artifacts into versioned archives
//   - Extract dependencies from compressed packages
//   - Automated compression before artifact upload
//
// Data Processing:
//   - Stream-process large datasets from compressed archives
//   - Convert between compression formats without recompression
//   - Transform data without intermediate files
//
// Web Services:
//   - On-the-fly compression of API responses
//   - Dynamic archive generation for downloads
//   - Streaming extraction of uploaded archives
//
// # Subpackages
//
// archive:
//
//	Multi-file container management with TAR and ZIP support.
//	- TAR: Sequential streaming (O(1) memory)
//	- ZIP: Random access (requires io.ReaderAt)
//	- Auto-detection via header analysis
//	- Unified Reader/Writer interfaces
//	- See: github.com/nabbar/golib/archive/archive
//
// compress:
//
//	Single-file compression with multiple algorithms.
//	- GZIP, BZIP2, LZ4, XZ support
//	- Magic number detection
//	- Streaming compression/decompression
//	- See: github.com/nabbar/golib/archive/compress
//
// helper:
//
//	High-level compression pipelines with thread-safe buffering.
//	- Unified interface for all operations
//	- Async I/O via goroutines
//	- Custom buffer preventing premature EOF
//	- See: github.com/nabbar/golib/archive/helper
//
// # Thread Safety
//
// All operations are thread-safe when used correctly:
//
//   - Compression/decompression: Each operation creates independent readers/writers
//   - Format detection: Stateless functions safe for concurrent use
//   - Extraction: Create separate instances for concurrent extractions
//   - Helper pipelines: Thread-safe with atomic operations and mutexes
//
// Concurrency Guidelines:
//   - Use separate instances per goroutine for archive readers/writers
//   - Format detection functions are safe for concurrent calls
//   - Helper package provides thread-safe buffering
//   - Verified with go test -race (zero data races)
//
// # Error Handling
//
// The package propagates errors from underlying operations:
//
//   - io.EOF: End of stream (expected)
//   - fs.ErrInvalid: Invalid input or corrupted data
//   - fs.ErrNotExist: File not found in archive
//   - Compression errors: Propagated from algorithm implementations
//
// Best Practice:
//   - Always check errors from all operations
//   - Use defer for resource cleanup
//   - Handle io.EOF appropriately for streaming
//
// # Best Practices
//
// Stream Large Files:
//
//	// ✅ Good: Streaming
//	func extract(path, dest string) error {
//	    in, err := os.Open(path)
//	    if err != nil {
//	        return err
//	    }
//	    defer in.Close()
//
//	    return archive.ExtractAll(in, path, dest)
//	}
//
//	// ❌ Bad: Load entire file
//	func extractBad(path, dest string) error {
//	    data, _ := os.ReadFile(path) // Full file in RAM!
//	    return processArchive(bytes.NewReader(data), dest)
//	}
//
// Always Close Resources:
//
//	// ✅ Good
//	reader, err := compress.Gzip.Reader(file)
//	if err != nil {
//	    return err
//	}
//	defer reader.Close() // Always close
//
// Handle Errors Properly:
//
//	// ✅ Good
//	n, err := io.Copy(dst, src)
//	if err != nil {
//	    return fmt.Errorf("copy failed: %w", err)
//	}
//
//	// ❌ Bad
//	io.Copy(dst, src) // Ignoring errors!
//
// # Limitations
//
//   - TAR: Sequential access only (no random file access)
//   - ZIP: Requires io.ReaderAt (seekable source)
//   - Compression levels: Not configurable (uses defaults)
//   - Encryption: Not supported (use external tools)
//
// # Dependencies
//
// Standard Library:
//   - io: Stream operations
//   - archive/tar: TAR format handling
//   - archive/zip: ZIP format handling
//   - compress/gzip: GZIP compression
//   - compress/bzip2: BZIP2 decompression
//
// External:
//   - github.com/pierrec/lz4/v4: LZ4 compression
//   - github.com/ulikunitz/xz: XZ compression
//
// # Testing
//
// The package includes comprehensive testing:
//   - 112+ test specifications using Ginkgo v2 + Gomega
//   - ≥80% code coverage (target met)
//   - Zero data races (verified with -race flag)
//   - Thread-safety validation
//   - Performance benchmarks
//
// Run Tests:
//
//	go test ./...                          # All tests
//	go test -cover ./...                   # With coverage
//	CGO_ENABLED=1 go test -race ./...     # Race detection
//
// See TESTING.md for detailed testing documentation.
//
// # Examples
//
// See example_test.go for complete runnable examples covering:
//   - Archive extraction with auto-detection
//   - Compression format detection
//   - Algorithm parsing and validation
//   - Error handling patterns
//
// # Related Packages
//
//   - github.com/nabbar/golib/size: Size constants (KiB, MiB, etc.)
//   - Standard library archive/tar and archive/zip
//   - Standard library compress packages
//
// # References
//
//   - GoDoc: https://pkg.go.dev/github.com/nabbar/golib/archive
//   - Repository: https://github.com/nabbar/golib
//   - Testing Guide: TESTING.md
//   - README: README.md
package archive