Hi Guys,

I have a new question related to TriggerScheduler. I am seeing very erratic behavior on datasets triggers. I ahve a Cron scheduler that creates a dataset after a file gets dropped on s3 into a project and some tags, in particular "processed=false".
I have a TriggerScheduler that has a add_dataset_trigger that triggers a task id. The Cron scheduler works great, but the TriggerScheduler, is 10% of the time maybe.
Here is my config:

trigger = TriggerScheduler(
        pooling_frequency_minutes=0.1, sync_frequency_minutes=0.1
    )
for client in batch_transfer_params:
  trigger.add_dataset_trigger(
            name=f"batch processing - {client['client_name']}",
            # schedule_function=trigger_dataset_func,
            schedule_task_id=schedule_task_id,
            schedule_queue="high-mem",
            trigger_project=client["client_name"],
            trigger_name="batch_processing - incoming",
            target_project=client["client_name"],
            task_parameters={
                "client_name": client["client_name"],
                "dataset_id": "${dataset.id}",
            },
            trigger_required_tags=["dm=true", "in=true", "processed=false"],
            single_instance=True,
        )

I have tried so many scenarios right now I don't know what to think anymore, I cannot get it to work reliably.

• sometimes removing the tag processed=false and putting back on will trigger but sometimes it won't
• I checked the triggers after adding them via trigger.get_triggers() and it looks fine it creates one trigger per subfolder.
  Any pointers is appreciated.

thanks a lot for all the work.