Problem

You are executing an Apache Spark job in Databricks that converts UTC timestamps to the Asia/Manila time zone. However, you notice a conversion discrepancy for timestamps before 1899-09-06. You expect a +08:00 offset but see -15:56:08. For timestamps specifically on 1899-09-06 you see offsets incorrect by one minute.

You see these differences when using Databricks SQL or Spark operations such as from_utc_timestamp() or AT TIME ZONE 'Asia/Manila'. 
 

Cause

This behavior is expected and historically accurate. It originates from the way Databricks and Apache Spark handle time zone conversions based on the IANA Time Zone Database (TZDB) and the Proleptic Gregorian calendar introduced in Spark 3.0.

For details, refer to the Time Zones >> Asia/Manila timezonedb documentation.

Because Databricks relies on TZDB for historical accuracy, Spark correctly applies the historical Local Mean Time (LMT) offsets when converting historic timestamps.
 

Solution

To override historical differences, use a strict +08:00 offset.

1. Set the session time zone to a fixed-offset zone. 

SET TIME ZONE 'Etc/GMT-8';

2. Replace references to Asia/Manila with Etc/GMT-8 in your SQL or Spark code. 

SELECT from_utc_timestamp('1753-01-01 00:00:00', 'Etc/GMT-8');

Or

SELECT TIMESTAMP '1753-01-01 00:00:00' AT TIME ZONE 'Etc/GMT-8';

If your data can accommodate the discrepancy from historical accuracy, continue using Asia/Manila as the session or conversion time zone.

SET TIME ZONE 'Asia/Manila';
SELECT from_utc_timestamp('1753-01-01 00:00:00', 'Asia/Manila');