Damn python. This actual wasted me 2 hours of time.

Why Python, why?
Anyone faced this?

@ewpratten
At least it should throw some syntax errors. Didn't expect this.

I'm sorry but... 2 hours?

@noctemx

I'm working huge list of JSON files,
Making pandas, dataframe, and then write to parquet.

Each time, it picks up a file and then process it. When I'm running manually line by line, everything is fine. ( Here I'm just verifying the data frame, not any lists and writing to parquet )

When I'm running the entire script it is failing. Lastly I decided to take a look at the declared variables, and I found this. :( :/

RTFM

If u work with json files, y not load them to see if they're valid?

@evilmupp3t
It is not a problem to explain 📣🔊🎹
It is pain to share.

I was having defined columns., Which I'll check whether the column is there in the dataframe or not.

So one of the defined column is supposed to be, 'seller_name' and 'adtracking', but because of the above mistake, it became
'seller_name_adtracking'. Which obviously doesn't exists in the dataframe which is throwing errors.

@Gowtham95india and if u call json.load(filename) to get your json as a python object, you don't have to read line by line in the first place.

@evilmupp3t

Why holding the data in another variable.
Import pandas as pd
f = 'file:///var/file.json'
df=pd.read_json(f, lines=True)