>Workloads and Roadmap

// What

Inference at request scale for ParalleliX AI: open-source models, each request served whole by one node. Distributed model training across heterogeneous GPUs and dataset preprocessing belong to the broader parallelizable-workload path.

// Sub-task shape

Inference (live): the inference request is the unit of work, dispatched whole to one node. Never split across machines. Training (general path): one sub-task per mini-batch replica (data parallelism), per layer shard (tensor parallelism), or per pipeline stage per micro-batch (pipeline parallelism).

// Segmentation

• Request-level inference (live)

  A single inference is not segmented. The whole request routes to one capable node; many requests run concurrently across the pool, so throughput scales with the number of online nodes.

• Data parallelism (training, general path)

  Identical model replicas process disjoint mini-batches; gradients aggregate at synchronisation barriers. One sub-task per replica per step.

• Tensor and pipeline parallelism (training, general path)

  Layer-level matrix multiplications shard across devices, and forward/backward passes split across pipeline stages. One sub-task per shard per layer, or per stage per micro-batch.

// Hardware fit

Tier 2 (mid-GPU) or tier 3 for inference depending on model size. Tier 3 (A100, H100) for training. Tier 1 (CPU) for preprocessing and tokenisation.

// Latency posture

Inference: latency-sensitive; each request is served whole, target p95 under 200ms per request. Training: latency-tolerant; steps take seconds to minutes.

// Failure + mitigation

Inference: a slow or dropped node stalls only the requests routed to it; re-dispatch the request whole to another capable node. Training (general path): synchronisation barriers couple nodes, so a slow node delays the whole step; mitigation is deadline-bounded sub-tasks plus re-dispatch on the slowest tail.