Resolved: How to copy all files and folders of a different azure directory using databricks (without Azure data factory)

In this post, we will see how to resolve How to copy all files and folders of a different azure directory using databricks (without Azure data factory)

Question:

i want to connect to folder in other azure directory from Databricks. I have access keys but it is not working. What is other way to do it?

spark.conf.set( "fs.azure.account.key.<storageAccountName>.dfs.core.windows.net","accessKey")

df=spark.read.csv("abfss://containerName@storageAccountName.dfs.core.windows.net/folder/")
display(df)

Error

AbfsRestOperationException: Operation failed: "Server failed to authenticate the request. Please refer to the information in the www-authenticate header.", 401, HEAD, https://<storageAccount>.dfs.core.windows.net/folder/?upn=false&action=getAccessControl&timeout=

Best Answer:

Changing cluster make same code worked. Not sure of the reason though

If you have better answer, please add a comment about this, thank you!

Source: Stackoverflow.com

Question:

Best Answer:

Related