🧩 Module System & Package Manager

Hierarchical Modules with Comprehensive Dependency Management

Overview

The STARK module system provides a hierarchical namespace organization with explicit import/export declarations, semantic versioning, and a powerful package manager for dependency resolution.

Key Features

  • Hierarchical Modules - Clean namespace organization with explicit boundaries
  • Semantic Versioning - Full SemVer support with sophisticated dependency resolution
  • Security-First - Package signing, vulnerability scanning, and supply chain security
  • Workspace Support - Multi-package projects with shared dependencies
  • Cross-Platform - Platform-specific dependencies and conditional compilation
  • Registry Ecosystem - Multiple registry support including private/corporate registries

Module System Overview

// Module declaration (optional for file-level modules)
module ml::vision::detection;

// Imports from standard library
use std::tensor::{Tensor, Device};
use std::dataset::{Dataset, DataLoader};
use std::model::{Model, Loss, Optimizer};

// Imports from external packages
use opencv::{Mat, imread, imwrite};
use torchvision::transforms::{Resize, Normalize};

// Imports from local modules
use super::backbone::{ResNet, Backbone};
use crate::utils::{Timer, Logger};

// Module contents
pub struct YOLODetector {
    backbone: Box,
    head: DetectionHead,
    anchors: Tensor
}

impl YOLODetector {
    pub fn new(num_classes: i32) -> Self { ... }
    pub fn detect(image: Tensor) -> [Detection] { ... }
    
    // Private function (not exported)
    fn nms(boxes: [BoundingBox], threshold: f32) -> [BoundingBox] { ... }
}

// Export specific items
pub use backbone::ResNet50;
pub use utils::Timer as PerfTimer;

Module Hierarchy and File Organization

Project Structure

src/
├── lib.stark              // Root module (defines crate public API)
├── ml/
│   ├── mod.stark          // ml module declaration
│   ├── vision/
│   │   ├── mod.stark      // ml::vision module
│   │   ├── detection.stark // ml::vision::detection
│   │   ├── classification.stark
│   │   └── segmentation.stark
│   ├── nlp/
│   │   ├── mod.stark
│   │   ├── tokenizers.stark
│   │   └── transformers.stark
│   └── audio/
│       ├── mod.stark
│       └── preprocessing.stark
├── utils/
│   ├── mod.stark
│   ├── logging.stark
│   └── metrics.stark
└── examples/
    ├── yolo_detection.stark
    └── sentiment_analysis.stark

Import and Export Syntax

Import Declarations

// Basic imports
use std::tensor::Tensor;                    // Import single item
use std::dataset::{Dataset, DataLoader};   // Import multiple items
use std::model::*;                          // Import all public items (discouraged)

// Qualified imports
use std::tensor;                            // Import module, use as tensor::Tensor
use std::dataset as data;                   // Import with alias

// Relative imports
use super::backbone::ResNet;                // Parent module
use crate::utils::Logger;                   // Crate root
use self::submodule::Helper;                // Current module's submodule

// Conditional imports (platform/feature specific)
#[cfg(feature = "gpu")]
use cuda::kernels::*;

#[cfg(target_os = "linux")]
use linux_specific::optimizations;

// External package imports
use numpy as np;                            // Python interop
use opencv::{Mat, Size};                    // C++ library binding
use @huggingface/transformers::{AutoModel, AutoTokenizer};  // External registry

// Version-specific imports
use torch@">=1.13,<2.0"::{nn, optim};     // Version constraints
use tensorflow@"2.12.0"::{keras};          // Exact version

// Optional imports with fallbacks
use gpu_accelerated::fast_conv2d catch std::tensor::conv2d;

Export Declarations

// Public exports (available to external crates)
pub fn train_model() -> Model { ... }
pub struct NeuralNetwork { ... }
pub enum ActivationFunction { ... }
pub const DEFAULT_LEARNING_RATE: f32 = 0.001;

// Conditional exports
#[cfg(feature = "experimental")]
pub fn experimental_feature() { ... }

// Re-exports from submodules
pub use vision::{Detection, Classification, Segmentation};
pub use nlp::transformers::{BERT, GPT, T5};

// Renamed exports
pub use internal::ComplexName as SimpleName;

// Module-level visibility modifiers
pub(crate) fn internal_api() { ... }        // Visible within crate
pub(super) fn parent_accessible() { ... }   // Visible to parent module
pub(in crate::ml) fn ml_only() { ... }     // Visible within ml module tree

Module Resolution Rules

Resolution order for use ml::vision::detection::YOLOv5:

  1. Current crate modules:
    src/ml/vision/detection.stark → YOLOv5
  2. Standard library:
    std::ml::vision::detection::YOLOv5
  3. External dependencies (Package.stark):
    ml-toolkit::vision::detection::YOLOv5
  4. Global package registry:
    @pytorch/vision::detection::YOLOv5

Disambiguation with explicit paths:

use crate::ml::vision::detection::YOLOv5;     // Local module
use std::ml::vision::detection::YOLOv5;       // Standard library
use pytorch_vision::detection::YOLOv5;        // External dependency
use @pytorch/vision::detection::YOLOv5;       // Registry package

Package.stark Manifest Format

Complete Package.stark Example

# Package.stark - Project manifest file
[package]
name = "stark-cv"
version = "0.3.1"
description = "Computer Vision library for STARK"
authors = ["AI Research Team "]
license = "MIT OR Apache-2.0"
homepage = "https://github.com/company/stark-cv"
repository = "https://github.com/company/stark-cv.git"
documentation = "https://docs.company.com/stark-cv"
readme = "README.md"
keywords = ["computer-vision", "ai", "ml", "deep-learning"]
categories = ["machine-learning", "computer-vision"]
edition = "2024"
rust-version = "1.70"  # Minimum STARK version

# Binary targets
[[bin]]
name = "stark-cv-cli"
path = "src/bin/cli.stark"
required-features = ["cli"]

# Library configuration
[lib]
name = "stark_cv"
path = "src/lib.stark"
crate-type = ["cdylib", "rlib"]  # Both dynamic and static linking

# Dependencies
[dependencies]
# Standard dependencies
stark-std = "1.0"
tensor-lib = "2.1.0"
dataset-lib = "1.8"

# External dependencies with version constraints
opencv = { version = "4.8", features = ["contrib", "opencv_dnn"] }
numpy = "1.24"
torch = { version = ">=2.0,<3.0", optional = true }

# Git dependencies
ml-commons = { git = "https://github.com/ml-commons/stark", branch = "main" }
internal-tools = { git = "ssh://git@internal.com/ml/tools.git", tag = "v1.2.0" }

# Path dependencies (local development)
shared-models = { path = "../shared-models" }

# Registry dependencies
transformers = { registry = "huggingface", version = "4.30" }
vision-models = { registry = "pytorch", version = "0.15.0" }

# Platform-specific dependencies
[target.'cfg(target_os = "linux")'.dependencies]
cuda-toolkit = "11.8"
tensorrt = { version = "8.6", optional = true }

[target.'cfg(target_os = "macos")'.dependencies]
metal-performance-shaders = "14.0"

[target.'cfg(target_arch = "wasm32")'.dependencies]
wasm-bindgen = "0.2"
web-sys = "0.3"

# Development dependencies (for tests, examples, benchmarks)
[dev-dependencies]
stark-test = "1.0"
criterion = "0.5"  # Benchmarking
proptest = "1.2"   # Property-based testing
tempfile = "3.0"

# Build dependencies (for build scripts)
[build-dependencies]
cmake = "0.1"
pkg-config = "0.3"
cc = "1.0"

Feature Flags

# Feature flags
[features]
default = ["std", "tensor-ops"]

# Core features
std = ["stark-std/std"]
no-std = []
tensor-ops = ["tensor-lib/ops"]

# Optional ML frameworks
pytorch = ["torch", "torchvision"]
tensorflow = ["tensorflow-rs"]
onnx = ["onnxruntime"]

# Hardware acceleration
gpu = ["cuda", "opencl"]
cuda = ["cuda-toolkit", "cuDNN"]
opencl = ["opencl-sys"]
metal = ["metal-performance-shaders"]
tensorrt = ["tensorrt-sys"]

# Optimization levels
fast = ["tensor-ops/simd", "gpu"]
experimental = ["unstable-features"]

# Python interoperability
python = ["pyo3", "numpy"]

# Web assembly support
wasm = ["wasm-bindgen", "web-sys"]

# CLI tools
cli = ["clap", "colored", "indicatif"]

Workspace Configuration

# Workspace configuration (for multi-package projects)
[workspace]
members = [
    "packages/stark-cv-core",
    "packages/stark-cv-models", 
    "packages/stark-cv-datasets",
    "packages/stark-cv-cli"
]

exclude = [
    "examples/*",
    "benchmarks/*"
]

resolver = "2"

# Workspace dependencies (shared across workspace members)
[workspace.dependencies]
stark-std = "1.0"
tensor-lib = "2.1.0"
serde = "1.0"

# Profile configurations
[profile.release]
opt-level = 3
debug = 0
debug-assertions = false
overflow-checks = false
lto = "fat"
panic = "abort"
codegen-units = 1
strip = "symbols"

# Custom profiles for specific use cases
[profile.gpu-optimized]
inherits = "release"
opt-level = 3
lto = "fat"
codegen-units = 1
target-cpu = "native"

[profile.memory-optimized]
inherits = "release"
opt-level = "s"  # Optimize for size
lto = "thin"
panic = "abort"

Dependency Management and Versioning

Semantic Versioning (SemVer) Support

# Version specification formats in Package.stark

# Exact version
opencv = "4.8.0"

# SemVer ranges
tensor-lib = "^2.1.0"    # >=2.1.0, <3.0.0 (compatible)
dataset = "~1.8.0"       # >=1.8.0, <1.9.0 (bugfix updates)
ml-utils = ">=1.5,<2.0"  # Range specification

# Pre-release versions
experimental-ai = "3.0.0-alpha.1"
beta-features = "2.5.0-beta"
nightly = "1.0.0-nightly.20241126"

# Version with build metadata
custom-build = "1.2.3+build.456"

# Git-based versioning
ml-research = { git = "https://github.com/research/stark-ml", rev = "abc123" }
internal = { git = "...", tag = "v2.1.0" }
latest = { git = "...", branch = "main" }

Lock File Format (Package.lock)

# Package.lock - Exact dependency tree for reproducible builds
# This file is automatically generated by stark package manager.
# Do not edit manually.

version = 3

[[package]]
name = "stark-cv"
version = "0.3.1"
dependencies = [
    "opencv",
    "tensor-lib", 
    "dataset-lib"
]

[[package]]
name = "opencv"
version = "4.8.1"
source = "registry+https://packages.stark-lang.org/"
checksum = "sha256:1234567890abcdef..."
dependencies = [
    "opencv-sys"
]

[[package]]
name = "tensor-lib"
version = "2.1.3"
source = "registry+https://packages.stark-lang.org/"
checksum = "sha256:abcdef1234567890..."
dependencies = [
    "blas-sys",
    "cuda-sys"
]

[[package]]
name = "ml-research"
version = "0.1.0"
source = "git+https://github.com/research/stark-ml#abc123def456"
dependencies = [
    "experimental-features"
]

# Package metadata and resolution information
[metadata]
resolver = "stark-resolver-v2"
generated-at = "2024-11-26T10:30:00Z"
stark-version = "1.0.0"

# Checksums for security verification
[metadata.checksums]
"opencv 4.8.1" = "sha256:1234567890abcdef..."
"tensor-lib 2.1.3" = "sha256:abcdef1234567890..."

Dependency Resolution Algorithm

Dependency resolution follows these principles:

  1. Semantic Version Resolution
    • Use highest compatible version within constraints
    • Prefer stable versions over pre-release
    • Respect minimum version requirements
  2. Conflict Resolution Strategy
    • Multiple versions of same package are not allowed
    • Use "minimal version selection" algorithm
    • Fail fast on incompatible constraints
  3. Feature Unification
    • Features are additive across the dependency graph
    • If any dependency enables a feature, it's enabled everywhere
    • Optional dependencies are only included if required
  4. Platform-Specific Resolution
    • Resolve platform-specific dependencies based on target
    • Support cross-compilation scenarios
    • Handle conditional dependencies correctly

Resolution Examples

// Example resolution process:
Package A requires: opencv ^4.8.0
Package B requires: opencv ~4.8.1  
Package C requires: opencv >=4.8.0,<4.9.0

Resolution: opencv 4.8.1 (highest compatible version)

// Conflict example:
Package A requires: tensor-lib ^2.0.0
Package B requires: tensor-lib ^3.0.0

Resolution: ERROR - No compatible version found

Package Manager Commands

Core Commands

Project Management

# Project initialization
stark new my-project                    # Create new project
stark new --lib my-library             # Create library project
stark new --bin my-binary              # Create binary project
stark init                             # Initialize in existing directory

Dependency Management

# Dependency management
stark add opencv@4.8                   # Add dependency
stark add --dev criterion              # Add dev dependency
stark add --optional pytorch           # Add optional dependency
stark add --features "gpu,cuda" tensor-lib  # Add with specific features
stark remove opencv                    # Remove dependency
stark update                           # Update all dependencies
stark update opencv                    # Update specific dependency

Building and Testing

# Building and testing
stark build                            # Build project
stark build --release                  # Release build
stark build --target wasm32-wasi       # Cross-compilation
stark test                             # Run tests
stark test --integration              # Run integration tests
stark bench                           # Run benchmarks
stark check                           # Check without building

Documentation

# Documentation
stark doc                             # Generate documentation  
stark doc --open                      # Generate and open docs
stark doc --no-deps                   # Skip dependency docs

Package Publishing

# Package publishing
stark login                           # Login to registry
stark logout                          # Logout from registry
stark publish                         # Publish to registry
stark publish --registry internal     # Publish to specific registry
stark yank 1.0.0                     # Yank published version
stark owner --add user@email.com      # Add package owner

Package Information

# Package information
stark search "computer vision"         # Search packages
stark info opencv                     # Package information
stark deps                           # Show dependency tree
stark deps --duplicates              # Show duplicate dependencies
stark licenses                       # Show dependency licenses
stark audit                          # Security vulnerability audit

Security and Trust

Package Verification

# Package signing and verification
[package.metadata.security]
signing-key = "ed25519:1234567890abcdef..."
require-signatures = true
trusted-publishers = [
    "stark-lang-team",
    "pytorch-team", 
    "nvidia"
]

# Security policies in .stark/config.toml
[security]
require-signatures = true
allow-git-dependencies = false
audit-on-install = true
max-package-size = "100MB"

blocked-packages = [
    "malicious-package@*",
    "vulnerable-lib@<1.2.0"
]

trusted-keys = [
    "ed25519:abcdef1234567890...",
    "rsa:fedcba0987654321..."
]

# Audit database integration
[audit]
database = "https://security.stark-lang.org/advisory-db"
ignore = [
    "STARK-2024-001",  # Ignore specific advisory
]

Supply Chain Security

Security Commands

# Security commands
stark audit                           # Check for vulnerabilities
stark audit --fix                     # Auto-fix vulnerable dependencies
stark verify                         # Verify package signatures
stark sbom                           # Generate Software Bill of Materials
stark license-check                   # Verify license compatibility

Example Security Audit Output

$ stark audit
    Checking 145 dependencies for security advisories
    Found 2 vulnerabilities:
    
    ┌─────────────────────────────────────────────────────────────┐
    │                        Advisory                              │
    │                        ━━━━━━━━                              │
    │ Package      ┆ tensor-lib                                    │
    │ Version      ┆ 2.0.1                                        │
    │ Advisory     ┆ STARK-2024-0123                              │
    │ Severity     ┆ High                                         │
    │ Description  ┆ Buffer overflow in tensor operations         │
    │ Solution     ┆ Upgrade to >= 2.0.2                         │
    └─────────────────────────────────────────────────────────────┘
    
    Run `stark audit --fix` to upgrade vulnerable dependencies.

Key Benefits

🏗️ Hierarchical Organization

Clear namespace management with explicit imports/exports and module boundaries

📦 Rich Package Management

Comprehensive Package.stark format supporting all modern package management features

🔄 Semantic Versioning

Full SemVer support with sophisticated dependency resolution and conflict handling

🛡️ Security First

Package signing, vulnerability scanning, and comprehensive supply chain security

🏢 Workspace Support

Multi-package projects with shared dependencies, metadata, and coordinated releases

🌐 Cross-Platform

Platform-specific dependencies, conditional compilation, and cross-compilation support

🏪 Registry Ecosystem

Multiple registry support with private/corporate registry integration and authentication

⚡ Developer Experience

Rich CLI commands, helpful tooling, and comprehensive package management workflows