AWS Data Pipelines - (Disaster) Recovery and Validation of pipeline-definitions using BOTO3

I have recently started to use AWS data-pipelines for backing up (cross-region) S3 buckets.

The BOTO3 Python API allows you to create, and validate an AWS data-pipeline, and apply a pipeline definition (which contains the low-level AWS configuration).  This definition is useful for date-pipeline validation, and can be applied to multiple pipelines.

I have taken the AWS JSon, which is available (once a pipeline has been created), and parsed this into pipelineObjects, parameterObjects, and parameterValues, which can be used to put_pipeline_definition, and validate_pipeline_definition (for existing data pipelines).

The Git repo is here.  Look in test_create_and_validate_pipeline_definition for how the JSon is converted into a pipeline definition, with a valid response (with no errors).


<span class="n">response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">validate_pipeline_definition</span><span class="p">(</span>
    <span class="n">pipelineId</span><span class="o">=</span><span class="s">'string'</span><span class="p">,</span>
    <span class="n">pipelineObjects</span><span class="o">=</span><span class="p">[</span>
        <span class="p">{</span>
            <span class="s">'id'</span><span class="p">:</span> <span class="s">'string'</span><span class="p">,</span>
            <span class="s">'name'</span><span class="p">:</span> <span class="s">'string'</span><span class="p">,</span>
            <span class="s">'fields'</span><span class="p">:</span> <span class="p">[</span>
                <span class="p">{</span>
                    <span class="s">'key'</span><span class="p">:</span> <span class="s">'string'</span><span class="p">,</span>
                    <span class="s">'stringValue'</span><span class="p">:</span> <span class="s">'string'</span><span class="p">,</span>
                    <span class="s">'refValue'</span><span class="p">:</span> <span class="s">'string'</span>
                <span class="p">},</span>
            <span class="p">]</span>
        <span class="p">},</span>
    <span class="p">],</span>
    <span class="n">parameterObjects</span><span class="o">=</span><span class="p">[</span>
        <span class="p">{</span>
            <span class="s">'id'</span><span class="p">:</span> <span class="s">'string'</span><span class="p">,</span>
            <span class="s">'attributes'</span><span class="p">:</span> <span class="p">[</span>
                <span class="p">{</span>
                    <span class="s">'key'</span><span class="p">:</span> <span class="s">'string'</span><span class="p">,</span>
                    <span class="s">'stringValue'</span><span class="p">:</span> <span class="s">'string'</span>
                <span class="p">},</span>
            <span class="p">]</span>
        <span class="p">},</span>
    <span class="p">],</span>
    <span class="n">parameterValues</span><span class="o">=</span><span class="p">[</span>
        <span class="p">{</span>
            <span class="s">'id'</span><span class="p">:</span> <span class="s">'string'</span><span class="p">,</span>
            <span class="s">'stringValue'</span><span class="p">:</span> <span class="s">'string'</span>
        <span class="p">},</span>
    <span class="p">]</span>
<span class="p">)</span>

Author | Miles Davenport

Career programmer, who designs, assembles, fixes, and supports customers, software and systems.