[Feature Request] Allow keys with `$` in them

[kenahoo]

🚀 Feature Request

Following the discussion at https://hydra-framework.zulipchat.com/#narrow/stream/213879-Hydra/topic/Override.20keys.20with.20'.24'.20in.20them , it would be great if the grammar could be tweaked so that $ is an allowed character in YAML keys when it's not part of an interpolation like ${foo}, since $ is a valid non-special character in YAML.

Motivation

Our application uses special keys like $args and $class with a prefix to distinguish them from the other data in the configuration. However, since these aren't allowed as keys in some contexts by Hydra, attempting to override them (e.g. at the command line) produces an error:

% python app.py 'agents.$args.c=11'
LexerNoViableAltException: agents.\$args.c=11
                                   ^
See https://hydra.cc/docs/next/advanced/override_grammar/basic for details

Pitch

In general, it seems like any valid YAML should be accepted by Hydra in its config files and override mechanisms, unless it's specifically in conflict with Hydra additions.

Describe alternatives you've considered

We could use a different prefix like _ or APP_ or the like, but we'd love to avoid that, since we already have substantial infrastructure in place with this syntax and it seems at first look to not actually conflict with the Hydra ${foo} syntax.

Another alternative would be to use Hydra's _target_ mechanism for instantiation; so far it looks like we couldn't do this because our in-house mechanism is a bit more full-featured for our specific use cases.

Are you willing to open a pull request? (See CONTRIBUTING)

I probably don't have enough expertise in the existing codebase to do so.

Additional context

[omry]

Hi @kenahoo.

In general, it seems like any valid YAML should be accepted by Hydra in its config files and override mechanisms, unless it's specifically in conflict with Hydra additions.

There is no such requirement. In fact - stating it like this will guarantee that it will make future addition to Hydra potential breaking changes to people relying on some obscure features of yaml.

We can take a look. To set expectations: This is hot a high pri request, if this turns out to be hard I don't think we should support it.
(I have no plans of officially supporting "any valid YAML" in keys, just to look into adding $ as a supported character there).

@odelalleau, can you take a look once you merge your outstanding PR for nested interpolations?
Since your change will move interpolation from a token to a parser rule it will definitely interact with this.
(I gave it a quick shot with the current grammar and not surprisingly adding $ to ID is breaking interpolation tests).

[kenahoo]

What I meant by "Hydra additions" is any current functionality and any reserved syntax for future functionality. If the set of reserved syntax can't be cleanly stated, it leaves users in much the same boat - not sure whether something will break in the future as Hydra gains features.

[omry]

Reserved syntax can't be stated because future functionality is not yet defined.
The safest course of action is to be restrictive and lessen restrictions over time instead of starting as wide as possible and then make things more restrictive as we run into edge cases.

If (when) a new feature of Hydra require breaking backward compatibility, effort will be invested into making sure the transition is as smooth as possible.
You can see examples of this in the Upgrade section in the website (last item on the sidebar).

[odelalleau]

@odelalleau, can you take a look once you merge your outstanding PR for nested interpolations?

Ok