We don't recommend using S3 lifecycle policies because, as you mentioned, it removes files without updating metadata and creates dangling references. In addition, it often doesn't implement the lifecycle policy you want because it removes files based on the modified time of the file and not on the data itself. If you compact, you reset the age used to trigger the policy even though the data hasn't changed. Instead, you should use a lifecycle policy on the data itself. Tabular, for example, has a service where you can set a maximum age for rows and select a column that holds the creation date. Then we automatically remove rows just like S3, but keeping metadata up to date.

@@tabularIO Thank you. Have a few other questions and thoughts, but this is not the forum for such. Will reach out over slack or whatever channel when applicable