Skip to content

/ spark

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE REQUEST]: Expose metadata #589

Open
thrixton opened this issue Jul 13, 2020 · 3 comments
Open

[FEATURE REQUEST]: Expose metadata #589

thrixton opened this issue Jul 13, 2020 · 3 comments
Labels
enhancement good first issue

Comments

@thrixton
Copy link
Contributor

@thrixton thrixton commented Jul 13, 2020

This is more a question than a feature request.

When parsing JSON files, I need to sanitize the field names so field with spaces becomes field_with_spaces.
I want to preserve the original name as well, metadata about the column if you like :)

There is a metadata field on StructField, but it is internal.
Why is this internal, is it possible or desirable to expose it?
@imback82
Copy link
Contributor

@imback82 imback82 commented Jul 13, 2020

Thanks @thrixton brining this up.

It is internal for now since it is not properly mapped to the Metadata class on the Scala side: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/Metadata.scala. If we want to expose this as public, we need to properly wrap the field.
@thrixton
Copy link
Contributor Author

@thrixton thrixton commented Jul 13, 2020

Good to know @imback82.
Currently, I'm covered as I just modify the JSON DOM directly and then parse it using StructType and have the internals visible so can access Metadata.
I guess at some point full support will go into the backlog.
@thrixton thrixton closed this Jul 13, 2020
@imback82
Copy link
Contributor

@imback82 imback82 commented Jul 13, 2020

Let me keep this open so we can follow up on this feature.
@imback82 imback82 reopened this Jul 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement good first issue
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
@thrixton @imback82
You can’t perform that action at this time.