Handling non-unique keys in objects

[yo35]

Hi,

I would like to re-open the discussion on how objects with non-unique should be handled at load. This topic has been discussed previously there:

• Non-unique keys in objects. #375
• Handling Duplicate keys in json #3392

As stated by @nlohmann, RFC 7159 states that:

When the names within an object are not
unique, the behavior of software that receives such an object is
unpredictable. Many implementations report the last name/value pair
only. Other implementations report an error or fail to parse the
object, and some implementations report all of the name/value pairs,
including duplicates.

... and currently, the nlohmann::json library implements the 1st behavior only (i.e. only the last key/value pair is loaded in case of duplicated keys).

@nlohmann suggests that the 2nd behavior (i.e. reporting an error) could be implemented using parser callbacks. However:

• implementing detection of duplicates is not that straightforward, especially when dealing with nested objects (see impl proposed by @jwestell: Non-unique keys in objects. #375 (comment)),
• reporting the location of the duplicates (line/column number in the JSON file) is not possible with the parser callback,
• I believe that preferring to report an error than silently ignore some of the key/value pairs is a quite common use-case, especially when loading JSON files that need thorough validation (typically because of possible manual edition).

Then, would it be possible to implement the "raise error in case of duplicate keys" behavior as a builtin optional feature in the nlohmann::json library? (typically a behavior for which it could be possible to opt-in when invoking nlohmann::json::parse(..))

[dingminghbut]

I think the answer is: yes, it would be reasonable as an opt-in parse mode, but today it probably needs a custom layer because the current DOM result cannot represent duplicate object names.

Once parsing has produced a json object, duplicates are already gone because the object storage maps each name to one value. So duplicate detection has to happen while parsing, before the object is materialized.

A parser callback can catch the common case by keeping a stack of key sets, one set per object depth, for example:

#include <nlohmann/json.hpp>
#include <stdexcept>
#include <string>
#include <unordered_set>
#include <vector>

std::vector<std::unordered_set<std::string>> keys;

auto reject_duplicate_keys = [&](int depth,
                                 nlohmann::json::parse_event_t event,
                                 nlohmann::json& parsed) {
    using event_t = nlohmann::json::parse_event_t;

    if (event == event_t::object_start) {
        if (keys.size() <= static_cast<std::size_t>(depth)) {
            keys.resize(static_cast<std::size_t>(depth) + 1);
        }
        keys[static_cast<std::size_t>(depth)].clear();
        return true;
    }

    if (event == event_t::key) {
        auto& seen = keys[static_cast<std::size_t>(depth)];
        const auto key = parsed.get_ref<const std::string&>();
        if (!seen.insert(key).second) {
            throw std::runtime_error("duplicate JSON object key: " + key);
        }
        return true;
    }

    if (event == event_t::object_end && keys.size() > static_cast<std::size_t>(depth)) {
        keys[static_cast<std::size_t>(depth)].clear();
    }

    return true;
};

const auto j = nlohmann::json::parse(input, reject_duplicate_keys);

That said, your two objections are valid:

• it is easy to get the stack/depth bookkeeping wrong for nested objects;
• the simple callback path does not give a nice built-in parse error with exact duplicate-key source location.

So if this were added to the library, I would expect it to live below the DOM-building layer, probably as a parser option or SAX-level behavior, not as a post-parse validation step. That would also allow the library to report the duplicate at the point where the key token is read.

For now, if you need strict validation for user-edited config files, I would wrap parsing in one small helper like the callback above and treat the thrown exception as a validation error. For high-quality diagnostics with positions, a SAX handler or a dedicated pre-validation parser would be a better fit than checking the completed json value.

[nlohmann]

Thanks @dingminghbut for the detailed workaround — it matches the direction we want to go: no library-code change here, since the parser-callback approach already covers this today.

We have documented it as a recipe instead: PR #5259. While writing it up as a compiled, tested example, we found and fixed an off-by-one bug in the depth-indexed bookkeeping from the snippet above — object_start reports the depth of the object's parent, while the key events inside that object report depth + 1, so indexing the per-depth key set with the same depth in both places causes an out-of-bounds access on nested objects. The corrected version in the PR handles nested and sibling objects correctly (verified against flat duplicates, nested duplicates, sibling objects sharing key names, and arrays of objects).

This comment was written by Claude Code on behalf of @nlohmann.