When using a map transformation on a RDD using Databricks Runtime 9.1 LTS and above, the resulting schema order is different when compared to doing the same map transformation using Databricks Runtime 7.3 LTS.
Databricks Runtime 9.1 LTS and above incorporate Apache Spark 3.x. Starting with Spark 3.0.0, rows created from named arguments do not have field names sorted alphabetically. Instead, they are ordered in as entered.
To enable Spark 2.x style row sorting set PYSPARK_ROW_FIELD_SORTING_ENABLED to true in your cluster's Spark config (AWS | Azure | GCP).
For Python versions less than 3.6, the field names can only be sorted alphabetically.