When using a map transformation on a RDD using Databricks Runtime 9.1 LTS and above, the resulting schema order is different when compared to doing the same map transformation using Databricks Runtime 7.3 LTS.
Databricks Runtime 9.1 LTS and above incorporate Apache Spark 3.x. Starting with Spark 3.0.0, rows created from named arguments do not have field names sorted alphabetically. Instead, they are ordered in as entered.
For Python versions less than 3.6, the field names can only be sorted alphabetically.