Performance

Compare Panache’s formatting, linting, and language-server performance with other Markdown and Quarto tools on individual documents, whole repositories, and editor sessions.

Overview

These benchmarks compare Panache’s formatting, linting, and language-server performance with other Markdown and Quarto tools. They measure individual documents, whole repositories, and editor sessions. Tools differ in supported syntax, formatting policy, lint coverage, and editor features, so timings should be read alongside the work each tool performs.

The formatting and linting comparisons cover Markdown and Quarto documents. The language-server comparisons use separate Markdown and Quarto workloads to measure startup, responses to editor requests, performance after edits, and memory use.

The benchmark suite also reports separate parsing and formatting times for individual documents.

The timing charts use a logarithmic scale in milliseconds; lower is faster. Formatting and linting charts show mean times, with colors identifying documents or repositories. Language-server charts show medians, with colors identifying measurements. Hover over a point for details. Tools are ordered from fastest to slowest on average in the formatting and linting charts.

data = FileAttachment("performance_data.json").json();

timeTicks = (times) => {
    const [min, max] = d3.extent(times);
    return d3
        .range(Math.floor(Math.log10(min)), Math.ceil(Math.log10(max)) + 1)
        .flatMap((exponent) => [1, 2, 5].map((value) => value * 10 ** exponent))
        .filter((value) => value >= min && value <= max);
};

documentMeta = new Map(data.documents.map((d) => [d.id, d]));

records = data.results.map((r) => ({
    formatter: r.formatter,
    document: r.document,
    documentName: documentMeta.get(r.document)?.name ?? r.document,
    mean_ms: r.mean_ms,
    runs: r.runs,
}));

formatters = Array.from(
    d3.rollup(
        records,
        (v) => d3.mean(v, (d) => d.mean_ms),
        (d) => d.formatter,
    ),
)
    .sort(([, a], [, b]) => a - b)
    .map(([k]) => k);

documentIds = data.documents.map((d) => d.id);

html`<div class="callout-note">
  <div class="callout-header" style="font-weight: 600">Test Environment</div>
  <ul style="margin: 0">
    <li>Generated: ${new Date(data.meta.generated_at).toUTCString()}</li>
    <li>Host: ${data.meta.host.cpu} (${data.meta.host.os}/${data.meta.host.arch})</li>
    <li>Minimum runs: ${data.meta.min_runs}</li>
    <li>panache ${data.meta.tools.panache.version}, prettier ${data.meta.tools.prettier?.version ?? "n/a"}, pandoc ${data.meta.tools.pandoc?.version ?? "n/a"}, rumdl ${data.meta.tools.rumdl?.version ?? "n/a"}, mdformat ${data.meta.tools.mdformat?.version ?? "n/a"}, yamark ${data.meta.tools.yamark?.version ?? "n/a"}, mado ${data.meta.tools.mado?.version ?? "n/a"}, markdownlint ${data.meta.tools.markdownlint?.version ?? "n/a"}, markdownlint-cli2 ${data.meta.tools["markdownlint-cli2"]?.version ?? "n/a"}</li>
  </ul>
</div>`;

Formatting

Single-Document

We compare panache format with Prettier, Pandoc, rumdl, mdformat, and Yamark on individual documents (Figure 1). Each command formats one document in a fresh process, so the timings include startup overhead.

Plot.plot({
    marginLeft: 60,
    marginRight: 20,
    marginBottom: 40,
    marginTop: 20,
    height: 360,
    x: { label: null, domain: formatters },
    y: {
        type: "log",
        tickFormat: d3.format(",~g"),
        ticks: timeTicks(records.map((d) => d.mean_ms)),
        label: "Mean time (ms, log scale; lower is faster)",
        grid: true,
    },
    color: {
        legend: true,
        domain: documentIds,
        tickFormat: (id) => documentMeta.get(id)?.name ?? id,
        scheme: "tableau10",
    },
    marks: [
        Plot.dot(records, {
            x: "formatter",
            y: "mean_ms",
            fill: "document",
            r: 6,
            fillOpacity: 0.85,
            stroke: "white",
            strokeWidth: 1,
            channels: {
                document: "documentName",
                "time (ms)": (d) => d.mean_ms.toFixed(1),
            },
            tip: true,
        }),
    ],
});

Figure 1: Mean formatting time per document. Each dot represents one document, identified by color.

prepareRepoSuiteRows = (data, toolLabels) => {
    const repoMeta = new Map(data.repos.map((r) => [r.id, r]));

    return data.results
        .filter((r) => !r.failed && r.mean_ms > 0)
        .map((r) => {
            const repo = repoMeta.get(r.repo);
            const repoLabel = repo?.name ?? r.repo;
            const toolLabel = toolLabels[r.tool] ?? r.tool;
            return {
                ...r,
                repoLabel,
                toolLabel,
                fileCount: repo?.file_count ?? null,
                totalBytes: repo?.total_bytes ?? null,
            };
        });
};

In the recorded runs, Yamark has the shortest mean time on every document, followed by Panache. The tools perform different work, including different handling of math; the methodology describes those differences.

Repository-Wide

These comparisons run each formatter once per repository, processing its tracked Markdown or Quarto files together. They capture the cost of formatting many files in a single invocation.

repoMarkdownFormat = FileAttachment(
    "performance_repo_markdown_format_data.json",
).json();

repoMarkdownFormatResults = prepareRepoSuiteRows(repoMarkdownFormat, {
    panache: "panache",
    prettier: "prettier",
    rumdl: "rumdl",
    mdformat: "mdformat",
    yamark: "yamark",
});
repoMarkdownFormatTools = Array.from(
    d3.rollup(
        repoMarkdownFormatResults,
        (v) => d3.mean(v, (d) => d.mean_ms),
        (d) => d.toolLabel,
    ),
)
    .sort(([, a], [, b]) => a - b)
    .map(([k]) => k);
repoMarkdownFormatRepos = repoMarkdownFormat.repos.map((r) => r.id);
repoMarkdownFormatRepoMeta = new Map(
    repoMarkdownFormat.repos.map((r) => [r.id, r]),
);

The Markdown comparison includes Panache, Prettier, rumdl, mdformat, and Yamark (Figure 2).

Plot.plot({
    marginLeft: 60,
    marginRight: 20,
    marginBottom: 40,
    marginTop: 20,
    height: 320,
    x: { label: null, domain: repoMarkdownFormatTools },
    y: {
        type: "log",
        tickFormat: d3.format(",~g"),
        ticks: timeTicks(repoMarkdownFormatResults.map((d) => d.mean_ms)),
        label: "Mean time (ms, log scale; lower is faster)",
        grid: true,
    },
    color: {
        legend: true,
        domain: repoMarkdownFormatRepos,
        tickFormat: (id) => repoMarkdownFormatRepoMeta.get(id)?.name ?? id,
        scheme: "tableau10",
    },
    marks: [
        Plot.dot(repoMarkdownFormatResults, {
            x: "toolLabel",
            y: "mean_ms",
            fill: "repo",
            r: 6,
            fillOpacity: 0.85,
            stroke: "white",
            strokeWidth: 1,
            channels: {
                repo: "repoLabel",
                tool: "toolLabel",
                files: (d) => d.fileCount,
                "time (ms)": (d) => d.mean_ms.toFixed(1),
            },
            tip: true,
        }),
    ],
});

Figure 2: Mean formatting time across tracked Markdown files. Each dot represents one repository and tool, with colors identifying repositories.

repoQuartoFormat = FileAttachment(
    "performance_repo_quarto_format_data.json",
).json();

repoQuartoFormatResults = prepareRepoSuiteRows(repoQuartoFormat, {
    panache: "panache",
    rumdl: "rumdl",
    yamark: "yamark",
});
repoQuartoFormatTools = Array.from(
    d3.rollup(
        repoQuartoFormatResults,
        (v) => d3.mean(v, (d) => d.mean_ms),
        (d) => d.toolLabel,
    ),
)
    .sort(([, a], [, b]) => a - b)
    .map(([k]) => k);
repoQuartoFormatRepos = repoQuartoFormat.repos.map((r) => r.id);
repoQuartoFormatRepoMeta = new Map(
    repoQuartoFormat.repos.map((r) => [r.id, r]),
);

The Quarto comparison includes Panache, rumdl, and Yamark, processing tracked .qmd files (Figure 3).

Plot.plot({
    marginLeft: 60,
    marginRight: 20,
    marginBottom: 40,
    marginTop: 20,
    height: 320,
    x: { label: null, domain: repoQuartoFormatTools },
    y: {
        type: "log",
        tickFormat: d3.format(",~g"),
        ticks: timeTicks(repoQuartoFormatResults.map((d) => d.mean_ms)),
        label: "Mean time (ms, log scale; lower is faster)",
        grid: true,
    },
    color: {
        legend: true,
        domain: repoQuartoFormatRepos,
        tickFormat: (id) => repoQuartoFormatRepoMeta.get(id)?.name ?? id,
        scheme: "tableau10",
    },
    marks: [
        Plot.dot(repoQuartoFormatResults, {
            x: "toolLabel",
            y: "mean_ms",
            fill: "repo",
            r: 6,
            fillOpacity: 0.85,
            stroke: "white",
            strokeWidth: 1,
            channels: {
                repo: "repoLabel",
                tool: "toolLabel",
                files: (d) => d.fileCount,
                "time (ms)": (d) => d.mean_ms.toFixed(1),
            },
            tip: true,
        }),
    ],
});

Figure 3: Mean formatting time across tracked Quarto files. Each dot represents one repository and tool, with colors identifying repositories.

Yamark and Panache also have the shortest mean times in the repository formatting runs, in that order. The gap between them varies across repositories. Hover over a point to see the file count as well as the timing.

Linting

We compare panache lint, rumdl check, mado check, markdownlint, and markdownlint-cli2. Each linter checks for a different set of problems, so these timings compare the tools’ respective checks.

Single-Document

lintSingleData = FileAttachment("performance_lint_single_data.json").json();

lintDocumentMeta = new Map(lintSingleData.documents.map((d) => [d.id, d]));
lintSingleRecords = lintSingleData.results.map((r) => ({
    tool: r.tool,
    document: r.document,
    documentName: lintDocumentMeta.get(r.document)?.name ?? r.document,
    mean_ms: r.mean_ms,
    runs: r.runs,
}));
lintTools = Array.from(
    d3.rollup(
        lintSingleRecords,
        (v) => d3.mean(v, (d) => d.mean_ms),
        (d) => d.tool,
    ),
)
    .sort(([, a], [, b]) => a - b)
    .map(([k]) => k);
lintDocumentIds = lintSingleData.documents.map((d) => d.id);

The individual-file comparison uses the same documents as the formatting benchmark (Figure 4). Each invocation starts a fresh process.

Plot.plot({
    marginLeft: 60,
    marginRight: 20,
    marginBottom: 40,
    marginTop: 20,
    height: 300,
    x: { label: null, domain: lintTools },
    y: {
        type: "log",
        tickFormat: d3.format(",~g"),
        ticks: timeTicks(lintSingleRecords.map((d) => d.mean_ms)),
        label: "Mean time (ms, log scale; lower is faster)",
        grid: true,
    },
    color: {
        legend: true,
        domain: lintDocumentIds,
        tickFormat: (id) => lintDocumentMeta.get(id)?.name ?? id,
        scheme: "tableau10",
    },
    marks: [
        Plot.dot(lintSingleRecords, {
            x: "tool",
            y: "mean_ms",
            fill: "document",
            r: 6,
            fillOpacity: 0.85,
            stroke: "white",
            strokeWidth: 1,
            channels: {
                document: "documentName",
                "time (ms)": (d) => d.mean_ms.toFixed(1),
            },
            tip: true,
        }),
    ],
});

Figure 4: Mean linting time per document. Each dot represents one document, identified by color.

Mado has the shortest mean time on five of the six documents in these runs; Panache has the shortest time on the remaining document. Both take longer on the Pandoc manual than on the other inputs.

Repository-Wide

The Markdown comparison runs the same five linters on the repositories used for formatting (Figure 5).

repoMarkdownLint = FileAttachment(
    "performance_repo_markdown_lint_data.json",
).json();

repoMarkdownLintResults = prepareRepoSuiteRows(repoMarkdownLint, {
    panache: "panache lint",
    rumdl: "rumdl check",
    mado: "mado check",
    markdownlint: "markdownlint",
    "markdownlint-cli2": "markdownlint-cli2",
});
repoMarkdownLintTools = Array.from(
    d3.rollup(
        repoMarkdownLintResults,
        (v) => d3.mean(v, (d) => d.mean_ms),
        (d) => d.toolLabel,
    ),
)
    .sort(([, a], [, b]) => a - b)
    .map(([k]) => k);
repoMarkdownLintRepos = repoMarkdownLint.repos.map((r) => r.id);
repoMarkdownLintRepoMeta = new Map(
    repoMarkdownLint.repos.map((r) => [r.id, r]),
);

Plot.plot({
    marginLeft: 60,
    marginRight: 20,
    marginBottom: 40,
    marginTop: 20,
    height: 300,
    x: { label: null, domain: repoMarkdownLintTools },
    y: {
        type: "log",
        tickFormat: d3.format(",~g"),
        ticks: timeTicks(repoMarkdownLintResults.map((d) => d.mean_ms)),
        label: "Mean time (ms, log scale; lower is faster)",
        grid: true,
    },
    color: {
        legend: true,
        domain: repoMarkdownLintRepos,
        tickFormat: (id) => repoMarkdownLintRepoMeta.get(id)?.name ?? id,
        scheme: "tableau10",
    },
    marks: [
        Plot.dot(repoMarkdownLintResults, {
            x: "toolLabel",
            y: "mean_ms",
            fill: "repo",
            r: 6,
            fillOpacity: 0.85,
            stroke: "white",
            strokeWidth: 1,
            channels: {
                repo: "repoLabel",
                tool: "toolLabel",
                files: (d) => d.fileCount,
                "time (ms)": (d) => d.mean_ms.toFixed(1),
            },
            tip: true,
        }),
    ],
});

Figure 5: Mean linting time across tracked Markdown files. Each dot represents one repository and tool, with colors identifying repositories.

Mado has the shortest mean time on each Markdown repository, followed by Panache. The Quarto comparison uses Panache and rumdl on tracked .qmd files (Figure 6).

repoQuartoLint = FileAttachment(
    "performance_repo_quarto_lint_data.json",
).json();

repoQuartoLintResults = prepareRepoSuiteRows(repoQuartoLint, {
    panache: "panache lint",
    rumdl: "rumdl check",
});
repoQuartoLintTools = Array.from(
    d3.rollup(
        repoQuartoLintResults,
        (v) => d3.mean(v, (d) => d.mean_ms),
        (d) => d.toolLabel,
    ),
)
    .sort(([, a], [, b]) => a - b)
    .map(([k]) => k);
repoQuartoLintRepos = repoQuartoLint.repos.map((r) => r.id);
repoQuartoLintRepoMeta = new Map(repoQuartoLint.repos.map((r) => [r.id, r]));

Plot.plot({
    marginLeft: 60,
    marginRight: 20,
    marginBottom: 40,
    marginTop: 20,
    height: 320,
    x: { label: null, domain: repoQuartoLintTools },
    y: {
        type: "log",
        tickFormat: d3.format(",~g"),
        ticks: timeTicks(repoQuartoLintResults.map((d) => d.mean_ms)),
        label: "Mean time (ms, log scale; lower is faster)",
        grid: true,
    },
    color: {
        legend: true,
        domain: repoQuartoLintRepos,
        tickFormat: (id) => repoQuartoLintRepoMeta.get(id)?.name ?? id,
        scheme: "tableau10",
    },
    marks: [
        Plot.dot(repoQuartoLintResults, {
            x: "toolLabel",
            y: "mean_ms",
            fill: "repo",
            r: 6,
            fillOpacity: 0.85,
            stroke: "white",
            strokeWidth: 1,
            channels: {
                repo: "repoLabel",
                tool: "toolLabel",
                files: (d) => d.fileCount,
                "time (ms)": (d) => d.mean_ms.toFixed(1),
            },
            tip: true,
        }),
    ],
});

Figure 6: Mean linting time across tracked Quarto files. Each dot represents one repository and tool, with colors identifying repositories.

Panache and rumdl have similar mean linting times on the Quarto repositories, and their ranking varies by repository.

Markdown Language Server

We compare Panache with Marksman by opening the five largest tracked Markdown files in a pinned revision of the Rust Book. Each session requests diagnostics and navigation features, then performs 1,000 reference-label edits. A definition request follows each edit to check that the server has observed the change.

The benchmark measures startup, responses to editor requests, performance after edits, and memory use across three fresh sessions per server. Marksman indexes the Markdown workspace eagerly, while Panache analyzes open documents and discovers project relationships lazily. The results reflect those different strategies.

lspMemory = FileAttachment("performance_lsp_memory_data.json").json();

lspMemoryMilestones = [
    { key: "baseline", label: "Baseline" },
    { key: "settled", label: "Files open" },
    { key: "edited", label: "After edits" },
    { key: "peak", label: "Peak" },
];

lspMemoryRows = lspMemory.servers.flatMap((server) =>
    lspMemoryMilestones.map((milestone) => ({
        server: server.label,
        milestone: milestone.label,
        rss: server.aggregate.milestones[milestone.key].rss_mb,
        pss: server.aggregate.milestones[milestone.key].pss_mb,
    })),
);

lspServers = lspMemory.servers.map((server) => server.label);

html`<div class="callout-note">
  <div class="callout-header" style="font-weight: 600">Test Environment</div>
  <ul style="margin: 0">
    <li>Generated: ${new Date(lspMemory.meta.generated_at).toUTCString()}</li>
    <li>Host: ${lspMemory.meta.host.cpu} (${lspMemory.meta.host.os}/${lspMemory.meta.host.arch}, ${lspMemory.meta.host.memory_gb} GiB RAM)</li>
    <li>Corpus: ${lspMemory.corpus.name} at ${lspMemory.corpus.revision.slice(0, 12)}; ${lspMemory.session.file_count} open files (${(lspMemory.session.opened_bytes / 1024).toFixed(1)} KiB)</li>
    <li>Workload: ${lspMemory.meta.edit_count.toLocaleString()} edits; ${lspMemory.meta.runs} fresh processes per server</li>
    <li>Warm requests: ${lspMemory.meta.latency_warmups} warmup rounds and ${lspMemory.meta.latency_runs} measured rounds per target in each process</li>
    <li>${lspMemory.servers.map((server) => `${server.label}: ${server.version.replace(/^panache /, "")}`).join(", ")}</li>
  </ul>
</div>`;

Startup and Edit Runtime

Figure 7 measures four stages:

• Initialize measures the time from process launch to the server’s response to the editor’s initialization request.
• Workspace ready measures the time from process launch until background work settles after initialization.
• Open files ready measures the time from opening the documents until diagnostics arrive, initial navigation requests finish, and background work settles.
• Edit workload measures the time to complete all reference-label edits and their definition requests.

Readiness estimates exclude the deliberate wait used to confirm that background work has settled. The methodology explains how the runner identifies these quiet periods.

lspRuntimeMetrics = [
    { key: "init_seconds", label: "Initialize" },
    { key: "workspace_ready_seconds", label: "Workspace ready" },
    { key: "documents_ready_seconds", label: "Open files ready" },
    { key: "edit_work_seconds", label: "Edit workload" },
];

lspRuntimeRows = lspRuntimeMetrics.flatMap((metric) =>
    lspMemory.servers.map((server) => ({
        server: server.label,
        row: metric.label,
        median: server.aggregate.timings[metric.key] * 1000,
        min: d3.min(server.runs, (run) => run[metric.key]) * 1000,
        max: d3.max(server.runs, (run) => run[metric.key]) * 1000,
    })),
);

lspRuntimeDomain = d3
    .scaleLog()
    .domain(d3.extent(lspRuntimeRows, (d) => d.median))
    .nice()
    .domain();

Plot.plot({
    width,
    marginLeft: 70,
    marginRight: 25,
    marginBottom: 50,
    height: 360,
    x: { label: null, domain: lspServers },
    y: {
        type: "log",
        tickFormat: d3.format(",~g"),
        ticks: timeTicks(lspRuntimeDomain),
        domain: lspRuntimeDomain,
        label: "Elapsed time (ms, log scale)",
        grid: true,
    },
    color: {
        legend: true,
        domain: lspRuntimeMetrics.map((d) => d.label),
        scheme: "tableau10",
    },
    marks: [
        Plot.dot(lspRuntimeRows, {
            x: "server",
            y: "median",
            fill: "row",
            r: 5,
            title: (d) =>
                `${d.row} — ${d.server}\nMedian: ${d.median.toFixed(2)} ms\nRange: ${d.min.toFixed(2)}–${d.max.toFixed(2)} ms`,
            tip: true,
        }),
    ],
});

Figure 7: Language-server startup and edit runtime on a logarithmic scale. Tools appear on the x-axis; colors identify runtime metrics. Dots show medians in milliseconds; tooltips include the minimum and maximum across fresh processes. Lower is faster.

Request Latency

After startup, we time requests for document symbols, hover information, definitions, references, and renaming. Figure 8 shows median response times, with the 95th percentile (p95) in tooltips. Edit to definition includes both the change notification and the following definition response, capturing the work needed to observe an edited document.

Tooltips also show result counts and sizes, which can differ between servers. Points marked “empty” returned no items in any measured request; their timings measure the overhead of responding without a result.

lspLatencyRows = lspMemory.servers[0].aggregate.request_latencies.flatMap(
    (request) =>
        lspMemory.servers.map((server) => {
            const result = server.aggregate.request_latencies.find(
                (r) => r.key === request.key,
            );
            return {
                ...result,
                server: server.label,
                row: request.label,
            };
        }),
);

lspLatencyTip = (d) =>
    `${d.row} — ${d.server}\nMedian: ${d.median_ms.toFixed(3)} ms\np95: ${d.p95_ms.toFixed(3)} ms\nSamples: ${d.samples}; empty: ${d.empty_results}\nReturned items: ${d.result_count_min}–${d.result_count_max}\nFiles involved: ${d.result_files_min == null ? "n/a" : `${d.result_files_min}–${d.result_files_max}`}\nMedian result size: ${d.payload_bytes_median} bytes`;

lspLatencyExtent = d3.extent(lspLatencyRows, (d) => d.median_ms);

lspLatencyDomain = d3.scaleLog().domain(lspLatencyExtent).nice().domain();

Plot.plot({
    width,
    marginLeft: 70,
    marginRight: 25,
    marginBottom: 50,
    height: 380,
    x: { label: null, domain: lspServers },
    y: {
        type: "log",
        tickFormat: d3.format(",~g"),
        ticks: timeTicks(lspLatencyDomain),
        domain: lspLatencyDomain,
        label: "Request latency (ms, log scale)",
        grid: true,
    },
    color: {
        legend: true,
        domain: lspMemory.servers[0].aggregate.request_latencies.map(
            (d) => d.label,
        ),
        scheme: "tableau10",
    },
    marks: [
        Plot.dot(lspLatencyRows, {
            x: "server",
            y: "median_ms",
            fill: "row",
            r: 5,
            title: lspLatencyTip,
            tip: true,
        }),
        Plot.text(
            lspLatencyRows.filter((d) => d.empty_results === d.samples),
            {
                x: "server",
                y: "median_ms",
                text: () => "empty",
                fill: "row",
                textAnchor: "start",
                dx: 7,
                dy: -10,
            },
        ),
    ],
});

Figure 8: Language-server request latency on a logarithmic scale. Tools appear on the x-axis; colors identify requests. Dots show medians in milliseconds; tooltips include p95. Lower is faster.

Warm symbol, definition, and reference requests have median response times below one millisecond for both servers in these runs. Requests after edits take longer, especially for Marksman. Panache’s hover and rename requests return empty results on this target, so their timings do not establish comparable feature performance.

Memory

Figure 9 shows median memory use across three fresh sessions, including child processes. RSS counts resident memory, including shared pages in each process. Tooltips also show PSS, which divides shared pages among the processes using them to estimate their share of physical memory.

Plot.plot({
    marginLeft: 55,
    marginRight: 20,
    marginBottom: 40,
    marginTop: 25,
    height: 320,
    x: { label: null },
    y: { label: "Median process-tree RSS (MB)", grid: true },
    fx: { label: null, domain: lspMemoryMilestones.map((d) => d.label) },
    color: { legend: true, domain: lspServers, scheme: "tableau10" },
    marks: [
        Plot.barY(lspMemoryRows, {
            fx: "milestone",
            x: "server",
            y: "rss",
            fill: "server",
            channels: { "PSS (MB)": "pss" },
            tip: true,
        }),
        Plot.text(lspMemoryRows, {
            fx: "milestone",
            x: "server",
            y: "rss",
            text: (d) => d.rss.toFixed(1),
            dy: -7,
        }),
    ],
});

Figure 9: Median whole-process-tree RSS during the language-server workload. Each milestone is measured after quiescence, except peak, which is the largest 150 ms sample from the complete session.

Panache uses less resident memory at every measured stage in these runs. This workload opens only five files from the project, so the difference should be read alongside the servers’ approaches to workspace indexing.

Quarto Language Server

We compare Panache with q2 lsp, the experimental Rust Quarto language server, using three authoring guides from a pinned q2 revision: computations, Markdown basics, and title blocks. The files live in a standalone project; neither server builds a whole-project index in this workload.

Each session opens the three files, requests their outlines, and performs 100 heading edits in the computations guide. This comparison covers diagnostics and document symbols; q2 does not implement the navigation features used in the Markdown comparison.

Panache returns diagnostics on request, while q2 sends them after processing an open or change notification. The timings include those different delivery paths and each server’s own checks.

quartoLsp = FileAttachment("performance_lsp_quarto_data.json").json();

quartoLspLatencies = quartoLsp.servers.flatMap((server) =>
    server.aggregate.request_latencies.map((metric) => ({
        server: server.label,
        metric: metric.label,
        median: metric.median_ms,
        p95: metric.p95_ms,
        samples: metric.samples,
        results: metric.result_count_median,
    })),
);

html`<div class="callout-note">
  <div class="callout-header" style="font-weight: 600">Test Environment</div>
  <ul style="margin: 0">
    <li>Generated: ${new Date(quartoLsp.meta.generated_at).toUTCString()}</li>
    <li>Host: ${quartoLsp.meta.host.cpu} (${quartoLsp.meta.host.os}/${quartoLsp.meta.host.arch}, ${quartoLsp.meta.host.memory_gb} GiB RAM)</li>
    <li>Corpus: ${quartoLsp.corpus.revision.slice(0, 12)}; ${quartoLsp.session.file_count} files (${(quartoLsp.session.opened_bytes / 1024).toFixed(1)} KiB)</li>
    <li>Workload: ${quartoLsp.meta.edit_count} edits; ${quartoLsp.meta.runs} fresh processes per server</li>
    <li>${quartoLsp.servers.map((server) => `${server.label}: ${server.version}; ${server.runs[0].diagnostic_mode} diagnostics`).join(". ")}</li>
  </ul>
</div>`;

Figure 10 shows median times from opening or editing a document to receiving diagnostics, along with response times for document-symbol requests. Tooltips include p95 and the number of diagnostics or symbols returned.

Plot.plot({
    width,
    height: 320,
    marginLeft: 70,
    marginRight: 30,
    marginBottom: 40,
    x: { label: null, domain: quartoLsp.servers.map((s) => s.label) },
    y: {
        type: "log",
        tickFormat: d3.format(",~g"),
        ticks: timeTicks(quartoLspLatencies.map((d) => d.median)),
        label: "Request latency (ms, log scale)",
        grid: true,
    },
    color: {
        legend: true,
        label: null,
        domain: ["Open to diagnostics", "Document symbols", "Edit to diagnostics"],
        scheme: "tableau10",
    },
    marks: [
        Plot.dot(quartoLspLatencies, {
            x: "server",
            y: "median",
            fill: "metric",
            r: 5,
            channels: {
                "p95 (ms)": "p95",
                samples: "samples",
                "results (median)": "results",
            },
            tip: true,
        }),
    ],
});

Figure 10: Quarto LSP latency in milliseconds on a logarithmic scale. Tools appear on the x-axis; colors identify requests. Dots show medians. Tooltips include p95, sample counts, and the median number of returned diagnostics or symbols. Lower is faster.

Panache has shorter median times for all three measurements in these runs. Both servers return outlines and report no error diagnostics, though their outlines contain different numbers of symbols.

The table below reports median initialization times and memory use across the three sessions. Panache uses less resident memory at each measured stage.

quartoLspResourceMetrics = [
    [
        "Initialize (ms)",
        (s) => (s.aggregate.timings.init_seconds * 1000).toFixed(2),
    ],
    ["Baseline RSS (MB)", (s) => s.aggregate.milestones.baseline.rss_mb],
    ["Files open RSS (MB)", (s) => s.aggregate.milestones.settled.rss_mb],
    ["After edits RSS (MB)", (s) => s.aggregate.milestones.edited.rss_mb],
    ["After edits PSS (MB)", (s) => s.aggregate.milestones.edited.pss_mb],
    ["Peak RSS (MB)", (s) => s.aggregate.milestones.peak.rss_mb],
];

html`<table class="table">
  <thead><tr><th>Metric</th>${quartoLsp.servers.map((s) => html`<th>${s.label}</th>`)}</tr></thead>
  <tbody>${quartoLspResourceMetrics.map(
        ([label, value]) =>
            html`<tr><td>${label}</td>${quartoLsp.servers.map((s) => html`<td>${value(s)}</td>`)}</tr>`,
    )}</tbody>
</table>`;

Methodology

Formatting and Linting

Individual-document comparisons start a fresh process for each file. Repository comparisons process tracked files in a single invocation per tool and repository. CLI caches are disabled. Failed repository runs have null timings in the data and are excluded from the plots.

Yamark runs with an empty configuration and external code formatters disabled (--config /dev/null --skip-embedded-formatters). Its default wrapping and style settings apply. Yamark uses source spans and formatting plans and preserves display math as opaque content. Panache builds the lossless syntax tree also used by its linter and language server, and parses and formats TeX math. These differences describe the work each formatter performs; the benchmarks do not isolate their individual costs.

Markdown Sessions

The runner starts three fresh processes per server in alternating order. Each session opens the same files, obtains diagnostics using the server’s advertised pull or push mode, primes navigation requests, and waits for background work to settle. It then times warm requests and the edit workload.

Requests run serially over stdio. Each target gets two unmeasured warmup rounds and 20 measured rounds per session. Medians and p95 pool the measured samples from all three sessions. Document-symbol requests cover the three largest open files. Hover, definition, references, and rename use the reference link appended to the largest file. References include the declaration; rename returns edits without applying them.

The edit workload performs 1,000 fixed-width reference-label renames per session, each followed by a definition request. Edit-to-definition timings include all 1,000 edits, with no discarded warmups. The total edit runtime excludes the final diagnostics and memory-settling wait. A failed or timed-out request fails the benchmark.

The runner samples the server and its descendants through Linux /proc every 150 ms. Background work has settled when CPU use stays below 5% of one core for five seconds. Workspace and open-file readiness timings end at the start of their respective quiet periods, confirmed only after the five seconds have elapsed. These estimates retain sampling error.

Memory measurements use the same sampler. Baseline, files-open, and after-edits measurements follow quiet periods. Peak is the largest sampled value during the complete session. The charts show medians across the three sessions; the JSON retains each session’s measurements.

Quarto Sessions

The runner starts three fresh sessions per server. Each session opens the files serially, measures document-symbol requests, and performs 100 full-document replacements that change an appended heading in the computations guide. Full replacements accommodate q2’s synchronization mode. A final outline request must contain the last edited heading.

File opens have no warmups. Symbol requests have two warmup rounds and 20 measured rounds per file per session. Latency medians and p95 pool samples across all three sessions. Each edit waits for diagnostics before the next begins. Panache uses pull diagnostics, with a request immediately after each open or change; q2 pushes diagnostics after processing the notification. This comparison does not measure Panache’s debounced push mode.

Both servers must accept every document without error diagnostics and return a nonempty outline. Empty diagnostic sets count as successful analyses; empty symbol results, protocol errors, and timeouts fail the benchmark.

Initialization measures process launch through the initialize response. Memory uses the same process-tree sampler and five-second quiet periods as the Markdown comparison. q2 performs open-document parsing and outline analysis; Panache also retains its syntax trees and analysis database.

Reproducibility

Run these commands from the repository root:

# Measure formatting on individual documents and whole repositories.
task bench:formatter

# Measure linting on individual documents and whole repositories.
task bench:linter

# Measure both language-server comparisons, including memory use.
task bench:lsp

The tasks download the benchmark documents and check that the required tools are available. The formatting and linting comparisons use hyperfine and jq. The language-server comparisons require Linux with /proc and Python 3, plus Marksman and q2 for their respective workloads. Outside the development environment, install Yamark and record its version:

uv tool install yamark==0.3.0
export PANACHE_BENCH_YAMARK_VERSION=0.3.0

The scripts in benches/ write the committed JSON files next to this document. These files supply the charts, machine details, and tool versions shown above. Previewing or rendering the documentation reads the saved results without running the benchmarks.

To refresh an individual comparison, run its script:

bash benches/compare_all.sh --json --out docs/guide/performance_data.json
bash benches/compare_lint_single.sh --out docs/guide/performance_lint_single_data.json
bash benches/compare_repo_suite.sh --mode format --track markdown --out docs/guide/performance_repo_markdown_format_data.json
bash benches/compare_repo_suite.sh --mode format --track quarto --out docs/guide/performance_repo_quarto_format_data.json
bash benches/compare_repo_suite.sh --mode lint --track markdown --out docs/guide/performance_repo_markdown_lint_data.json
bash benches/compare_repo_suite.sh --mode lint --track quarto --out docs/guide/performance_repo_quarto_lint_data.json
bash benches/compare_lsp_memory.sh
bash benches/compare_lsp_quarto.sh

The Quarto script downloads a pinned q2 release and verifies its SHA-256 hash, or uses the executable specified by Q2_BIN.

After refreshing the JSON, clear this page’s freeze cache and re-render it to pick up the new results:

rm -rf docs/_freeze/guide/performance
quarto render docs/guide/performance.qmd