The example you gave deals with two practices.

  1. We should not do processing with airflow executor. Executor should simply start the process on a dedicated system. Since you mentioned copying data from MySql to Object store, this type of task can be done by Sqoop service on an cluster.
  2. What I mentioned is that suppose we have two tasks. Second task needs data from the first task, in this case, we should not pass the data itself, rather store that intermediate data outside airflow system and pass the reference. Xcoms store in the airflow metadat db. If we directly pass data between tasks using xcoms it will overload the metadata db.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store