Advancing Spark - Managing Files with Unity Catalog Volumes

Рет қаралды 6,825

Күн бұрын

Пікірлер: 11

@vincentdelbaen8815 Жыл бұрын

Thank you sir! I'll try it out right away and probably include it to our ways of working. I feel it can reduce the burden and avoid creating external locations for each data analysts projects.

@datawithabe Жыл бұрын

Great video,! as always, best place to learn new Databricks features :)

@coleb1567 Жыл бұрын

Great video. One unrelated question: how do you guys manage deployments with databricks? I come from an airflow +Jenkins background as an engineer. Would you recommend Jenkins for databricks deployments?

@mc1912 Жыл бұрын

I remember Simon mentioning they use Terraform for infrastructure deployment, but maybe he can tell us more 😅

@user-em7kw5lf4f 11 ай бұрын

Love your work Simon. Do you know if it is possible to have a credential that is not associated with same cloud provider as the Unity Catalogue instance? I have Databricks environment deployed on Azure but one of the ingestions is via an S3 bucket. I would love to be able to set this up as an external volume.

@nachetdelcopet 3 ай бұрын

I think you will need to create a Access Conector in your AWS, then go to your Databricks workspace and create the storage credentials using the AWS Access Conector ID. Then you can replicate everything he has explained in the video for AWS

@ErikParmann 10 ай бұрын

So with mounts we can have the dev workspace mount the dev containers, and the prod environment mount the prod containers, and they both get mounted to the same path. So the notebook don't have to 'know' if its running in dev or prod. How will that work in this new world? I noticed that the path contains "dev". Does each notebook have to figure out what environment it is in, and then read/write from the right paths and catalogs based on some string manipulation?

@neelred10 6 ай бұрын

Exactly my thought. Maybe environment variable can store dev/qa/prod value and use it to dynamically generate path string.

@atulbansal8041 Жыл бұрын

How can I get the access of data ricks environment for learning. I know there is a community edition available but somehow I am not able to load my raw files into that