Troubleshooting Databricks Connect for Scala

Note

This article covers Databricks Connect for Databricks Runtime 13.3 LTS and above.

This article provides troubleshooting information for Databricks Connect for Scala. Databricks Connect enables you to connect popular IDEs, notebook servers, and custom applications to Databricks clusters. See What is Databricks Connect?. For the Python version of this article, see Troubleshooting Databricks Connect for Python.

Error: StatusCode.UNAVAILABLE, StatusCode.UNKNOWN, DNS resolution failed, or Received http2 header with status 500

Issue: When you try to run code with Databricks Connect, you get an error messages that contains strings such as StatusCode.UNAVAILABLE, StatusCode.UNKNOWN, DNS resolution failed, or Received http2 header with status: 500.

Cause: Databricks Connect cannot reach your cluster.

Solutions:

  • Check to make sure that your workspace instance name is correct. If you use environment variables, check to make sure the related environment variable is available and correct on your local development machine.

  • Check to make sure that your cluster ID is correct. If you use environment variables, check to make sure the related environment variable is available and correct on your local development machine.

  • Check to make sure that your cluster has the correct custom cluster version that is compatible with Databricks Connect.

The filename, directory name, or volume label syntax is incorrect on Windows

Issue: You are using Databricks Connect on Windows and see:

The filename, directory name, or volume label syntax is incorrect.

Cause: Databricks Connect was installed into a directory with a space in your path.

Solution: You can work around this by either installing into a directory path without spaces, or configuring your path using the short name form.

Error: Failed to initialize MemoryUtil

Issue: When you try to build a DatabricksSession, it returns an error Failed to initialize MemoryUtil.

Cause: Apache Arrow is a dependency of the Databricks Connect client, and it is trying to access a private Java method using reflection, which is by default blocked in Java 17 because of security considerations.

Solution:

Set the following JVM field prior to JVM initialization:

--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED

For more information, see Apache Arrow Java Compatibility.

Tip

To set Java options in IntelliJ, see IntelliJ Run/debug configuration.