Problem

When retrieving SQL query statement text from a system table, the Databricks REST API, or the Databricks SDK, you notice that the statement text for large SQL queries is truncated.

Cause

Queries that read from the statement_text column, the REST API query_text column, or make calls to the SDK with the query_history.list() method all depend on the backing system table system.query.history and the statement_text column. This table and column are subject to storage limits. When you exceed the storage limit, the text returned is truncated. 

Solution

Be aware of the system table limit when performing analysis on query history through the query history system table and corresponding REST API and SDK interfaces. Note that the exact text limit varies depending on factors such as on-disk storage compression. 

For more information, review the Query history system table reference (AWS | Azure | GCP) documentation.