Optimize open models for production - RunInfra

RunInfraby RightNow

Dashboard Sign in Get started

Backed by

Combinator

Paste a model. RunInfra benchmarks the options and picks the winner. Deploy it, or own the stack.

Example workloads

Every optimization ends with a result you can inspect and run.

You get a benchmark receipt and a runnable deployment kit. Nothing hidden.

Serving engine

Compared, not assumed

GPU target

Sized to the model

p95 latency

Benchmarked

Throughput

Measured per GPU

VRAM

Checked for fit

Cost

Tracked per run

GPU kernels

Tuned where supported

Deployment kit

Run it or export it

From prompt to a production stack you own

RunInfra compares, tunes, and benchmarks the stack. Deploy or export it.

Describe a Llama 3.1 70BQwen 2.5 7BDeepSeek V3Mistral 7BPhi-4Gemma 2 9BMixtral 8x7BWhisper Large V3 workload in plain English.

RunInfra compares vLLMSGLangTensorRT-LLMvLLM-Omni and every other engine your model can run on.

It tunes speculative decodingkernel generationserver tuningquantizationKV cache reuseFlashAttention v2continuous batching where it helps, with no config to hand-write.

Deploy on NVIDIA H100NVIDIA H200NVIDIA B200NVIDIA A100NVIDIA L40SNVIDIA L4 and pay per million tokens, or export the stack and self-host.

Deploy on thehardware you choose

Compare real GPU prices and deploy wherever fits. The deployment kit and your own cloud mean no lock-in, no rewrites.

Common questions

Can't find what you're looking for? Get in touch

What is RunInfra?

Describe what you want to run. RunInfra picks compatible open models, benchmarks GPUs, tunes the runtime, and gives you a deploy-ready stack.

Deploy your first optimized model, measured before you ship

Describe the goal. RunInfra builds and optimizes the stack.

Start Building View Pricing

End-to-end encryption

Isolated GPU infrastructure

Zero data retention

SOC 2 Type II

From prompt to a production stack you own

RunInfra compares, tunes, and benchmarks the stack. Deploy or export it.

Describe a Llama 3.1 70B workload in plain English.

RunInfra compares SGLang and every other engine your model can run on.

It tunes server tuning where it helps, with no config to hand-write.

Deploy on NVIDIA A100 and pay per million tokens, or export the stack and self-host.

New Session

Optimize Llama-3.1-8B-Instruct on vLLM for cheapest GPU with latency and VRAM checks

Capturing cost-first intent for Llama 3.1 8B on vLLM.

Intake updatedModel, engine, goal

ModelLlama 3.1 8B

EnginevLLM

GoalLow cost, latency checked

Plan drafted3.1s

10 execution phases prepared

Plan readyreview / 10 phases / ~23m

Optimize Llama 3.1 8B on vLLM for cheapest GPU with latency checks

Recommended path: vLLM on L4. Review the plan before execution.

Working...

Runbook generated from the workload

Llama 3.1 8B on vLLM, lowest viable cost

draft

Latency target

p95 under 60ms

VRAM budget

24 GB

Est. runtime

~23 min

Execution plan10 phases, 3 validated

01

AWQ int4 quantizationready

Weight-only int4, calibrated offline

02

FlashAttention v2ready

Fused attention kernels

03

Continuous batchingqueued

In-flight request scheduling

04

Paged KV cachequeued

fp8 cache in paged blocks

05

CUDA graph capturequeued

Replay the decode-step graph

06

Speculative decodingqueued

Draft model proposes tokens

07

Prefix cachingqueued

Reuse shared prompt prefixes

08

Tensor-parallel sizingready

Single GPU, no sharding

09

Warmup and autotunequeued

Lock kernel shapes pre-serve

10

Serving-config tunequeued

Batch size and concurrency

Review the plan, then run.vLLM, L4 to A100

Deploy or export the stack

Pick a target for the optimized L40S build.

ready

TargetSupported GPUs

RunInfra Cloud

selected

Fully managed endpoint, billed per million tokens

H100L40SA100L4

Your RunPod

Deploy to your own RunPod account

H100A100RTX 4090L40S

Modal

Serverless deploy on RunInfra Modal

H200H100A100L40S

Your Modal

Deploy to your own Modal workspace

H100A100L40ST4

Generated stackDockerfileserve.shruninfra.yaml

1FROM runinfra/vllm:0.6.3-l40s

2ENV MODEL=Llama-3.1-8B-Instruct

3ENV QUANTIZATION=awq_marlin

4COPY ./weights /models

5CMD vllm serve $MODEL \

6--quantization awq_marlin \

7--kv-cache-dtype fp8 \

8--enable-prefix-caching \

9--max-num-seqs 256 \

10--gpu-memory-utilization 0.92

$0.12 / 1M38ms p95L40S

Own the winning stack before you ship

Not a black box. You get the measured stack to run, deploy, or export.

Why owning your AI stack matters

Data privacy and control

Keep sensitive workloads on infrastructure you choose.

Customization

Tune the model, runtime, and GPU to your workload.

Performance ownership

Real tuning, measured, not assumed.

Portability

Run it on our cloud, or export it to yours.

Supported across the stack

Open models, serving engines, GPUs, and the clouds you deploy to. RunInfra supports every layer of the inference stack.

Models

LLMLlama 3.3ASRWhisperImageQwen-ImageEmbedNV-EmbedTTSParler-TTSLLMQwen2.5VideoCosmosVisionPixtralEmbedEmbeddingGemmaClassifyRoBERTaLLMDeepSeek-V3ImageSanaASRParakeetLLMMistralVideoWan 2.1EmbedGTEVisionQwen2-VLTTSMMS-TTSRerankQwen3 RerankerLLMGemma 2AudioMusicGenVisionDeepSeek-VL2LLMNemotronTTSFastPitchVisionPaliGemmaRerankNV-RerankQAAudioQwen2-AudioLLMHermes 3ASRCanaryClassifyBERTVisionLlama 3.2 VisionEmbedQwen3 EmbeddingLLMLlama 3.3ASRWhisperImageQwen-ImageEmbedNV-EmbedTTSParler-TTSLLMQwen2.5VideoCosmosVisionPixtralEmbedEmbeddingGemmaClassifyRoBERTaLLMDeepSeek-V3ImageSanaASRParakeetLLMMistralVideoWan 2.1EmbedGTEVisionQwen2-VLTTSMMS-TTSRerankQwen3 RerankerLLMGemma 2AudioMusicGenVisionDeepSeek-VL2LLMNemotronTTSFastPitchVisionPaliGemmaRerankNV-RerankQAAudioQwen2-AudioLLMHermes 3ASRCanaryClassifyBERTVisionLlama 3.2 VisionEmbedQwen3 Embedding

Engines

EnginevLLMEngineSGLangEngineTensorRT-LLMEnginevLLM OmniEngineTEIEngineTransformersEnginevLLMEngineSGLangEngineTensorRT-LLMEnginevLLM OmniEngineTEIEngineTransformersEnginevLLMEngineSGLangEngineTensorRT-LLMEnginevLLM OmniEngineTEIEngineTransformersEnginevLLMEngineSGLangEngineTensorRT-LLMEnginevLLM OmniEngineTEIEngineTransformersEnginevLLMEngineSGLangEngineTensorRT-LLMEnginevLLM OmniEngineTEIEngineTransformersEnginevLLMEngineSGLangEngineTensorRT-LLMEnginevLLM OmniEngineTEIEngineTransformers

GPUs

Clouds