AWS Glue PySpark: Change Column Data Types
Рет қаралды 3,382
Жыл бұрынThis video is about how to change column data types in AWS Glue using PySpark. This tutorial will walk through how to achieve this using the resolveChoice method in a dynamic frame
Code Example: github.com/AdrianoNicolucci/d...
#aws #awsglue #pyspark
@harishtripathi7273 Жыл бұрын
Thanks @DataEng Uncomplicated Very Informative Videos!! could you also please create a video to move Pyspark dynamic transformed Data Frame to Redshift Table
@ryanalex98 Жыл бұрын
Interesting video, thanks! I was under the impression that resolveChoice could only be used on the "ChoiceType" schema in AWS Glue? I've been using applyMapping to resolve this issue (which requires generating a mapping for all columns you wish to keep, not just the ones you want to change... tedious for frames with lots of columns!)
@DataEngUncomplicated Жыл бұрын
I know I thought this earlier too, this saves a lot of time
@KyuGShim Жыл бұрын
I saw aws glue does not support binary type. But my situation is I have to do ETL job from mongodb which some of column's type is UUID by binary. is there any chance that i can change to useable type in glue?
@DataEngUncomplicated 4 ай бұрын
Yes, you can handle binary UUID types in AWS Glue. While AWS Glue doesn’t natively support the binary data type, you can work around this by converting the binary UUIDs to a string representation in your ETL script. I found this good discussion about this issue: stackoverflow.com/questions/63470718/how-to-work-with-binary-or-uuid-types-inside-mongo-server-side-javascript
Cloud Quick Labs
Рет қаралды 7 М.