Resolved: Databricks Delta Live Table – How To Simply Append A Batch Source To a DLT Table?

Question:

Using Python and all the relevant DLT properties within Databricks, does anyone know how to simple append to a DLT table from a batch source?
In PySpark you can just use df.write.format(“delta”).mode(“append”) but since dlt requires you to return a Spark dataframe within its decorator, we can’t use the Dataframe Writer API.
Thanks in advance.

Answer:

Delta Live Tables has a notion of a streaming live table that is append-only by default. You can define your pipeline as triggered, that will be equivalent of the the Trigger.Once. Something like that:
@dlt.table
def append_only():
  return spark.readStream.format("xyz").load()
Here we use readStream just to make sure that when we run the pipeline again we won’t append the same content again & again.

If you have better answer, please add a comment about this, thank you!

If you like this answer, you can give me a coffee by <a href=”https://violencegloss.com/mkw7pgdwwr?key=2a96ade6f3760c7e63343a320febb78e”>click here</a>

Source: Stackoverflow.com