Fixing a Promptfoo result collision bug with duplicate provider ids

I fixed a bug I found while using Promptfoo, so this note summarizes what happened.

The PR is here.
promptfoo PR #10208

Background

I was evaluating whether a normal text response or a JSON response would produce better output, using a configuration like this.

prompts:
  - |
    Answer in one short sentence.
    Topic: {{topic}}
 
providers:
  - id: openai:chat:gpt-5-mini
    label: "plain text"
    config:
      temperature: 0
      max_tokens: 24
 
  - id: openai:chat:gpt-5-mini
    label: "json format"
    config:
      temperature: 1
      max_tokens: 96
      response_format:
        type: json_schema
        json_schema:
          name: custom_name
          strict: true
          schema:
            type: object
            additionalProperties: false
            required:
              - message
            properties:
              message:
                type: string
                description: message
 
tests:
  - vars:
      topic: why reproducible eval results matter
    assert:
      - type: contains-any
        value:
          - eval
          - reproducible
          - consistency
 
  - vars:
      topic: why model configuration should be visible in comparison views
    assert:
      - type: contains-any
        value:
          - configuration
          - comparison
          - visible

When I checked the results, I expected to see one column for the text output pattern and another for the JSON output pattern. Instead, only one side appeared because the results collided.

In LLM evaluation, it is common to compare the same model with different settings. For example, you might compare different temperature values. Promptfoo's documentation also describes this use case.
Promptfoo evaluate-llm-temperature

There was a workaround: adding label to each provider made the results display correctly.

providers:
  - id: openai:chat:gpt-5-mini
    label: "plain text"
    config:
      max_tokens: 128
 
  - id: openai:chat:gpt-5-mini
    label: "json format"
    config:
      max_tokens: 128
      response_format:
      # response_format details omitted

Still, it took me a while to notice the workaround. I felt Promptfoo should either warn when the same provider id is used more than once, or display the results correctly without requiring a label.

That is what this PR fixed.

What Was Happening

Promptfoo assigns a prompt index to each provider/prompt combination so it can store evaluation results in a table.

For example, consider a case where two providers return the same provider id, and there is only one prompt.

Even when the provider id and prompt are the same, there are two providers, so the evaluation table needs two result columns.

However, the previous implementation used a key like this to find which prompt/provider column a result should be written to.

`${provider}:${promptId}` -> index

In this case, both providers returned the same provider id, and the prompt was also the same, so the Map key became identical.

JavaScript Map cannot hold multiple values for the same key, so the second value overwrote the first one.

As a result, the Map looked like this.

The relevant code was evaluator.ts#L2044.

Because the required index = 0 information was lost, the first provider's evaluation result was also associated with the second column.

The Fix

After the fix, the Map stores an array of indices for the same key instead of a single index. commit 4043eca

`${provider}:${promptId}` -> [index1, index2, ...]

In the example above, the Map becomes:

This allows Promptfoo to keep every corresponding column index even when the same provider id and prompt combination appears more than once.

However, changing the value to an array is not enough on its own. Promptfoo still needs to know whether the provider currently being evaluated should use [0, 1][0] or [0, 1][1].

To solve that, the implementation counts how many times the same provider key has appeared during evaluation.

In code, the occurrence count is managed with a Map, and the prompt index matching that occurrence index is used.

const promptIndices = promptIndexMap.get(`${providerKey}:${promptId}`);
const promptIdx = promptIndices?.[providerOccurrenceIndex];

This made it possible to store each evaluation result in a separate column, even when the same provider id and prompt combination appeared multiple times.

Maintainer Follow-Up

I opened the PR with the fix above, and Michael from the Promptfoo maintainer team added a further improvement.

My initial fix kept the providerKey:promptId key and changed the value from a single index to an array of indices.

`${provider}:${promptId}` -> [index1, index2, ...]

That follow-up went one step further: it stopped looking up result columns from providerKey:promptId altogether.

Instead, when prompt columns are created, the implementation now keeps the columns owned by each provider directly.

Conceptually, the structure looks like this.

In other words, rather than finding columns later with a key, each provider receives its own promptIdx at the time the columns are created.

type ProviderColumns = {
  provider: ApiProvider;
  columns: { promptIdx: number; prompt: Prompt }[];
};

With this shape, evaluation can simply process columnsByProvider in order.

for (const { provider, columns } of columnsByProvider) {
  for (const { promptIdx, prompt } of columns) {
    // Store this provider/prompt result in the promptIdx column.
  }
}

The important point is that the execution plan now receives the actual columns that were created, instead of reconstructing them from display or identifier strings such as provider id or prompt id.

This makes the behavior more robust for cases like these.

The resume metrics path was also updated.

When metrics are restored from existing evaluation results, duplicate providers can resolve to the same stored prompt. If multiple columns share the same metrics object, updating one result can affect the other.

To avoid that, existing metrics are cloned with structuredClone, so each column gets independent metrics.

metrics: existingPrompt?.metrics
  ? structuredClone(existingPrompt.metrics)
  : createDefaultPromptMetrics()

My first fix avoided the collision by allowing multiple indices for the same key. Michael's follow-up went further and changed the design so result columns are no longer reconstructed from string keys that can collide.

Summary

The results are now displayed correctly.

This is separate from the main topic of this article, but while working on this fix, I also found and fixed an error in the documentation. promptfoo PR #10209

Promptfoo is very useful for prompt evaluation, so I have been using it quite a lot recently.
This was a small fix, but I am glad I was able to contribute in some way.