Question:Using Python and all the relevant DLT properties within Databricks, does anyone know how to simple append to a DLT table from a batch source?
In PySpark you can just use df.write.format(“delta”).mode(“append”) but since dlt requires you to return a Spark dataframe within its decorator, we can’t use the Dataframe Writer API.
Thanks in advance.
Answer:Delta Live Tables has a notion of a streaming live table that is append-only by default. You can define your pipeline as triggered, that will be equivalent of the the
Trigger.Once. Something like that:
Here we use
@dlt.table def append_only(): return spark.readStream.format("xyz").load()
readStreamjust to make sure that when we run the pipeline again we won’t append the same content again & again.
If you have better answer, please add a comment about this, thank you!
If you like this answer, you can give me a coffee by <a href=”https://violencegloss.com/mkw7pgdwwr?key=2a96ade6f3760c7e63343a320febb78e”>click here</a>