Thank you sir! I'll try it out right away and probably include it to our ways of working. I feel it can reduce the burden and avoid creating external locations for each data analysts projects.
@datawithabe Жыл бұрын
Great video,! as always, best place to learn new Databricks features :)
@coleb1567 Жыл бұрын
Great video. One unrelated question: how do you guys manage deployments with databricks? I come from an airflow +Jenkins background as an engineer. Would you recommend Jenkins for databricks deployments?
@mc1912 Жыл бұрын
I remember Simon mentioning they use Terraform for infrastructure deployment, but maybe he can tell us more 😅
@user-em7kw5lf4f11 ай бұрын
Love your work Simon. Do you know if it is possible to have a credential that is not associated with same cloud provider as the Unity Catalogue instance? I have Databricks environment deployed on Azure but one of the ingestions is via an S3 bucket. I would love to be able to set this up as an external volume.
@nachetdelcopet3 ай бұрын
I think you will need to create a Access Conector in your AWS, then go to your Databricks workspace and create the storage credentials using the AWS Access Conector ID. Then you can replicate everything he has explained in the video for AWS
@ErikParmann10 ай бұрын
So with mounts we can have the dev workspace mount the dev containers, and the prod environment mount the prod containers, and they both get mounted to the same path. So the notebook don't have to 'know' if its running in dev or prod. How will that work in this new world? I noticed that the path contains "dev". Does each notebook have to figure out what environment it is in, and then read/write from the right paths and catalogs based on some string manipulation?
@neelred106 ай бұрын
Exactly my thought. Maybe environment variable can store dev/qa/prod value and use it to dynamically generate path string.
@atulbansal8041 Жыл бұрын
How can I get the access of data ricks environment for learning. I know there is a community edition available but somehow I am not able to load my raw files into that