GiganticTurtle0
I'm assuming here that self.dask_client.map(read_and_process_file, filepaths) actually does the multi process/node processing. The way it needs to work, it has to store the current state of the process and then restore it on any remote node/process. In practice this means pickling the local variables (Task included).
First I would try to use a standalone static function for the map, DASK might be able to deduce it does not need to pickle anything, as it is standalone.
Another option is not to create the Task in the main script but on the class itself.
Third, and that is on us, maybe we could add _ getstate _/setstate to Task, so we can pickle t, where the state is just the Task ID, that wouild actually relatively easy to setup, what do you think?