Metadata-Version: 2.1
Name: airflow-multi-dagrun
Version: 2.3.1
Summary: MultiDagRunPlugin for airflow
Home-page: https://github.com/mastak/airflow_multi_dagrun
Author: Ihor Liubymov
Author-email: infunt@gmail.com
License: UNKNOWN
Description: [![Build Status](https://travis-ci.com/mastak/airflow_multi_dagrun.svg?branch=master)](https://travis-ci.com/mastak/airflow_multi_dagrun)
        
        # Multi dag run
        
        This plugin contains operators for triggering a DAG run multiple times
        and you can dynamically specify how many DAG run instances create.
        
        It can be useful when you have to handle a big data and you want to split it
        into chunks and run multiple instances of the same task in parallel.
        
        When you see a lot launched target DAGs you can set up more workers.
        So this makes it pretty easy to scale.
        
        ## Install
        
        ```bash
        pip install airflow_multi_dagrun
        ```
        
        ## Example
        
        Code for scheduling dags
        
        ```python
        import datetime as dt
        from airflow import DAG
        
        from airflow_multi_dagrun.operators import TriggerMultiDagRunOperator
        
        
        def generate_dag_run():
            for i in range(100):
                yield {'index': i}
        
        
        default_args = {
            'owner': 'airflow',
            'start_date': dt.datetime(2015, 6, 1),
        }
        
        
        dag = DAG('reindex_scheduler', schedule_interval=None, default_args=default_args)
        
        
        ran_dags = TriggerMultiDagRunOperator(
            task_id='gen_target_dag_run',
            dag=dag,
            trigger_dag_id='example_target_dag',
            python_callable=generate_dag_run,
        )
        ```
        
        This code will schedule dag with id `example_target_dag` 100 times and pass payload to it.
        
        
        Example of triggered dag:
        
         ```python
        dag = DAG(
            dag_id='example_target_dag',
            schedule_interval=None,
            default_args={'start_date': datetime.utcnow(), 'owner': 'airflow'},
        )
        
        
        def run_this_func(dag_run, **kwargs):
            print("Chunk received: {}".format(dag_run.conf['index']))
        
        
        chunk_handler = PythonOperator(
            task_id='chunk_handler',
            provide_context=True,
            python_callable=run_this_func,
            dag=dag
        )
        ```
        
        ## Run example
        There is docker-compose config, so it requires docker to be installed: `docker`, `docker-compose`
        1. `make init` - create db
        2. `make add-admin` - create `admin` user (is asks a password)
        3. `make web` - start docker containers, run airflow webserver
        4. `make scheduler` - start docker containers, run airflow scheduler
        
        `make down` will stop and remove docker containers 
        
        ## Contributions
        If you have found a bug or have some idea for improvement feel free to create an issue
        or pull request.
        
        ## License
        Apache 2.0
        
Platform: UNKNOWN
Requires-Python: >=3.6.0
Description-Content-Type: text/markdown
