My Battle with Datetimes in Prod (opens in new tab)

(datacompose.io)

5 pointstccole3mo ago2 comments

2 comments

tccoleOP3mo ago

I spent 8 hours debugging datetime parsing in a PySpark migration. Every solution failed… built-in functions returned silent NULLs, UDFs took 6 hours to run, pandas UDFs still failed on 8% of rows.

abstractspoon3mo ago

>> Date inference is genuinely hard—there’s no universal standard

Umm, ISO is the universal standard

j / k navigate · click thread line to collapse

2 comments

tccoleOP3mo ago

I spent 8 hours debugging datetime parsing in a PySpark migration. Every solution failed… built-in functions returned silent NULLs, UDFs took 6 hours to run, pandas UDFs still failed on 8% of rows.

abstractspoon3mo ago

>> Date inference is genuinely hard—there’s no universal standard

Umm, ISO is the universal standard

j / k navigate · click thread line to collapse