Create AWS EMR Cluster Using AWS CLI and Submit job

Рет қаралды 1,470

Жыл бұрын

Configuring EMR
Creating Cluster using CLI
You can refer below GitHub link
github.com/saurabhgarg013/cre...
In this Video , create and run an EMR cluster using AWS CLI.
I have covered below topics.
Create an IAM user
Configure and install AWS CLI
Create an EMR cluster using AWS CLI
Create Pyspark code or Job
Submit job using AWS CLI
Submit job using Primary node
Check output in s3 bucket
I have also created videos related to AWS EMR.
Create AWS EMR cluster using AWS management console and submit job
• AWS EMR Cluster Create...
AWS EMR theory
• AWS EMR Tutorial | Ama...
DIFF AWS GLUE VS EMR
• DIFF AWS GLUE VS EMR| ...
Command for creating cluster using AWS CLI
You can also refer doc for commands
github.com/saurabhgarg013/cre...
aws emr create-cluster --name MyEMRCluster --use-default-roles --release-label emr-6.11.0 --instance-count 3 --instance-type m5.xlarge --applications Name=Spark Name=Hadoop --ec2-attributes SubnetIds=subnet-03xxxx64,KeyName=monu --log-uri s3://myemrproject/logs/
Execute job using awl cli step
aws emr add-steps --cluster-id j-GAVB3ZN07CUB --steps Type=Spark,Name="MySparkJob",ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,--master,yarn,--conf,spark.yarn.submit.waitAppCompletion=true,s3://myemrproject/scripts/mypysparkscript_1.py]

Пікірлер: 7

@electricalsir 4 күн бұрын

thanks

@sakshiwakode 9 ай бұрын

🙏🙏👌tysm sir

@kashifsadiq4145 2 ай бұрын

Make a playlist about dataops in hindi thanks

@waseemswork 11 ай бұрын

Hi Saurabh, Thanks for the this tutorial and other tutorials which are very helpful. If time permit, could you please explain, how can we automate creating EMR cluster creation for spark jobs through Cloud Formation Templates ?

@isharkpraveen 5 ай бұрын

unable to install pip install awscli

@rahulpamnani6574 Жыл бұрын

An error occurred (ValidationException) when calling the RunJobFlow operation: Invalid InstanceProfile: EMR_EC2_DefaultRole.

@TechnoDevs Жыл бұрын

Thanks for watching video. it's looked your EMR_EC2_DefaultRole" is invalid. I think you tried to modified this policies. please review the changes that you made. or what you can do delete existing one and try to recreate it.