Benchmarks & Research
Evidence first. Results only when they exist.
An open benchmark programme for AI infrastructure researchers, hardware makers, model-serving teams, hyperscalers and universities.
SimulatedMeasured (lab)Independently reproduced
01Experiments
Planned experiments
No measured results have been published yet. Values appear here only once they exist, with provenance.
| ID | Experiment | Metric | Hardware | Runtime | Result | Status |
|---|---|---|---|---|---|---|
| exp-001 | Quantisation vs quality for isiZulu & Swahili | Model quality preservation | NVIDIA L40S | vLLM | Not yet measured | Planned |
| exp-002 | Decode throughput across heterogeneous accelerators | Tokens per second | H100 / MI300X / M2 Ultra | vLLM, llama.cpp | Not yet measured | Planned |
| exp-003 | Power-aware scheduling under load-shedding | Cost per successful task | NVIDIA L40S | vLLM | Not yet measured | Planned |
| exp-004 | Connectivity-loss failover to local execution | Outage recovery time | Jetson AGX Orin | llama.cpp | Not yet measured | Planned |
| exp-005 | Energy efficiency by precision | Tokens per joule | NVIDIA A100 | vLLM | Not yet measured | Planned |
| exp-006 | Tail latency under bursty demand | P50 / P95 / P99 latency | NVIDIA H100 | vLLM | Not yet measured | Planned |
| exp-007 | Runtime compatibility matrix | Hardware compatibility | 8 profiles | vLLM, Ollama, llama.cpp | Not yet measured | Planned |
02Benchmark categories
What we will measure
Tokens per secondtok/s
Time to first tokenms
P50 / P95 / P99 latencyms
Requests per secondreq/s
Cost per successful taskUSD
Tokens per jouletok/J
GPU utilisation%
Memory efficiency%
Outage recovery times
Network resilience% success
Model quality preservationΔ score
Hardware compatibilitypass/fail
03Methodology & reproducibility
How results will be produced
MeshAI Benchmark Methodology (draft v0.1) 1. Scope: inference workloads only; hardware, runtime, model, precision and driver versions recorded. 2. Warm-up: discard first 60 s of each run. 3. Load: fixed and bursty request profiles; input/output token distributions published. 4. Metrics: tok/s, TTFT, P50/P95/P99, req/s, cost per successful task, tok/J, utilisation, memory efficiency, outage recovery time, network resilience, quality preservation. 5. Energy: measured at the wall where possible; otherwise accelerator telemetry, labelled as such. 6. Quality: task-level evaluation, including African-language test sets, before and after optimisation. 7. Provenance: every result labelled Simulated, Measured (lab) or Independently reproduced. 8. Reproducibility: configs, seeds and scripts published with each result.
Note · Want to reproduce or contribute an experiment? Research partners can request access through the contact page.
More useful AI from every watt, GPU and dollar.
MeshAI is building the intelligence that makes AI infrastructure more efficient, resilient and economically accessible — starting where infrastructure constraints are most demanding.