jq - combine JSON objects with their arrays without duplicating or removing existing

Question

I'm trying to combine the arrays of two JSON files I need to achieve this by combining the arrays without removing existing values of arrays and without removing any object. Sorting output arrays would be nice but not necessary.

file-1.json:

{
  "title": {
    "message": {
      "receiver": [
        "one",
        "two",
        "three"
      ],
      "sender": [
        "two",
        "one",
        "one"
      ]
    }
  }
}

file-2.json:

{
  "title": {
    "message": {
      "receiver": [
        "four",
        "two",
        "three"
      ],
      "sender": [
        "one",
        "one",
        "one"
      ]
    }
  }
}

What I have tried:

$ jq -s 'add' file-1.json file-2.json

{
  "title": {
    "message": {
      "receiver": [
        "four",
        "two",
        "three"
      ],
      "sender": [
        "one",
        "one",
        "one"
      ]
    }
  }
}

Expected output:

{
  "title": {
    "message": {
      "receiver": [
        "one",
        "two",
        "three",
        "four"
      ],
      "sender": [
        "one",
        "two"
      ]
    }
  }
}

Kusalananda · Accepted Answer · 2025-12-12 12:44:37Z

The following command iterates over the path expressions for each array in the first file. It updates the arrays at those paths by adding the corresponding array from the second file. The combined array is uniquified.

$ jq 'input as $b | reduce paths(arrays) as $p (.; setpath($p; getpath($p) + ($b | getpath($p)) | unique))' file-1.json file-2.json
{
  "title": {
    "message": {
      "receiver": [
        "four",
        "one",
        "three",
        "two"
      ],
      "sender": [
        "one",
        "two"
      ]
    }
  }
}

This assumes:

There are two files.
Both files have the same overall structure, at least with regard to where the arrays are located.
No array contains other arrays. It'll still (possibly) work, but the sub-arrays will be merged too, which may be unexpected.

The jq expression, a bit more nicely written, with comments:

# Read the second file into the variable $b.
input as $b |

# Use reduce to iterate over all paths that lead to arrays
# in the first file. $p represents a path, and '.' is the 
# accumulator (starts as the first file).
reduce paths(arrays) as $p (.;

    # Update the value at path $p.
    setpath($p;

        # Get the array at path $p from the 1st file
        # and merge it with the array at the same path
        # in $b, the 2nd file.
        getpath($p) + ($b | getpath($p)) |

        # Remove duplicates from the merged array (will sort it).
        unique
    )
)

You are a life saver. I have previously seen your other answer here: https://unix.stackexchange.com/questions/706593/how-to-combine-json-objects-with-the-same-key-using-jq but this answer can be only used for objects. Thank you! — Zero
– Zero, Commented 9 hours ago

Stéphane Chazelas · Accepted Answer · 2025-12-12 19:50:17Z

Except for the deduplicating of array values, that merging of structures is exactly what the Hash::Merge perl module does.

You can however modify its merge algorithm to specify how you want two objects of a given type to be merged. So here, you could do:

perl -MList::Util=uniq -MJSON::PP -MHash::Merge -0777 -ne '
  BEGIN {
    # jq-like pretty-printing:
    $j = JSON::PP->new->indent->indent_length(2)->space_after->canonical;

    # change the LEFT_PRECEDENT behaviour (the one used by default),
    # so that when merging an array with an array, the result is the
    # deduplicated list of the union of the elements, instead of just
    # the concatenation of the two lists:
    Hash::Merge::get_behavior_spec("LEFT_PRECEDENT")
      ->{ARRAY}->{ARRAY} = sub { [uniq @{$_[0]}, @{$_[1]}] };
  }

  # Merge each object from each file in turn into $out
  $out = Hash::Merge::merge($out // {}, $j->decode($_));

  END {
    print $j->encode($out)
  }' file-*.json

Which on your two files gives:

{
  "title": {
    "message": {
      "receiver": [
        "one",
        "two",
        "three",
        "four"
      ],
      "sender": [
        "two",
        "one"
      ]
    }
  }
}

The List::Util and JSON::PP modules come with perl, but Hash::Merge you may have to install separately (libhash-merge-perl package on Debian and derivatives).

Stack Exchange Network

jq - combine JSON objects with their arrays without duplicating or removing existing

2 Answers 2

You must log in to answer this question.

Linked

Hot Network Questions

jq - combine JSON objects with their arrays without duplicating or removing existing

2 Answers 2

You must log in to answer this question.

Linked

Related

Hot Network Questions