Problem

When using the COPY INTO command to load data into a Unity Catalog table, you notice new data added to the source file is not copying into the table.

Cause

The COPY INTO command is designed to be idempotent, meaning that files in the source location that have already been loaded are skipped, even if the files have been modified since they were loaded. This is true even if you modify the table (for example, removing all rows) or modify the file.

Solution

Include the force option in the COPY_OPTIONS parameters and set it to true. This disables idempotency and force file loading regardless of whether they were previously loaded. 

COPY INTO <your-catalog>.<your-schema>.<your-table>
FROM '<your-data-source>'
FILEFORMAT = <your-data-source-file-format>
FORMAT_OPTIONS (
  'header' = 'true',
  'inferSchema' = 'true'
)
COPY_OPTIONS (
  'mergeSchema' = 'false',
  'force' = 'true'
);

For more information, please refer to the COPY INTO (AWS | Azure | GCP) documentation.