pipeline-graph-architecture

Pipeline Graph Architecture

The Core Model

Pipelines are graph traversals. Nodes are stages. Edges are transitions with optional middleware. Context accumulates as you traverse.

Node Structure

Every pipeline stage node has three layers:

# pqis-gradle-builder-myapp-12345

## Context

```yaml dynamo:pipeline-12345
\{\{include:pipeline-12345:context\}\}
```

## Mapping

```yaml
repository: ${context.repositories.backend}
repository_path: ${context.paths.gradle_root}
gradle_args: ${context.build.gradle_command}
```

## Build

\{\{include:pqis-gradle-builder:pattern\}\}

Layer 1: Context (Shared State)

DynamoDB-backed YAML fence
One record per pipeline run or artifact
Aliased via include from canonical location
All stages in a pipeline share the same context

Layer 2: Mapping (Wiring)

Created by IDE or pipeline generator
Maps raw context paths → pattern-expected locals
Pure YAML, no code
This is what you edit to configure a stage

Layer 3: Pattern (Reusable Template)

Include of the actual fences that do work
References ${local:mapping.field}
Same pattern used across all instances
Developer maintains once, used everywhere

Edge Structure

Edges connect nodes with optional middleware and conditions:

slots:
- slug: deploy-stage
  via:
    - fence: validate-artifacts
      args: {require: [docker-image]}
    - fence: security-scan
    - fence: audit-log
  context: []
  conditions:
    all:
      - type: pipelineEnvironment
        args:
          executeFor: [prod]

Via Chains

Fences that execute during transition:

Receive current context
Can transform/enrich it
Run in sequence
Can themselves be full subgraph traversals

Conditions

Multiple edges from same node, conditions determine which fire:

Evaluated against current context
Standard condition types (environment, artifact exists, etc.)
Same resolver pattern as conditions.groovy

Context Accumulation

The artifact/session IS the context record. Everything stacks onto it:

created: 2024-12-13T07:00:00Z

source:
  repository: git@github
  commit: abc123
  branch: main

builds:
  devl:
    timestamp: 2024-12-13T07:05:00Z
    executor: gradle-builder-arm64-v2.1
    outputs:
      jar: s3://artifacts/myapp-1.2.3.jar
      docker: ecr://myapp:1.2.3-devl

  preprod:
    timestamp: 2024-12-13T08:00:00Z
    promoted_from: devl
    outputs:
      docker: ecr://myapp:1.2.3-preprod

  prod:
    timestamp: 2024-12-13T12:00:00Z
    promoted_from: preprod
    stack: arn:aws:cloudformation:us-east-1:123:stack/myapp-prod
    service: arn:aws:ecs:us-east-1:123:service/myapp-prod

tags:
  - environment:prod
  - release:1.2.3
  - team:platform

Key Points

One record per artifact version - not scattered stores
Environments stack - devl, preprod, prod all on same record
Tags for querying - find all prod deployments, all v1.2.x, etc.
Full provenance - source, executors, outputs, everything

Pipeline Execution

The executor is simple:

1. Load current node
2. Execute all fences (they read/write context)
3. Look at edges (south, east, etc.)
4. Evaluate conditions against context
5. For matching edges:
   a. Run via chain (middleware)
   b. Recurse on target node
6. Context bubbles back up when branches complete

Example Flow

[Start] 
    ↓ south
[git-clone-myapp-123]     → produces: repository, commit
    ↓ south  
[gradle-builder-myapp-123] → requires: repository → produces: jar, docker-image
    ↓ south (condition: env=devl)     ↓ east (condition: env=prod)
[deploy-devl-myapp-123]              [security-scan-myapp-123]
    ↓ south                               ↓ south
[End]                                [deploy-prod-myapp-123]
                                          ↓ south
                                     [End]

Each node:

Reads what it needs from context
Does its work (fences execute)
Writes outputs to context
Traversal continues based on conditions

Input Stage: The Copier

Starting a pipeline run:

Fresh Start

input:
  type: fresh
  # Empty context, just source info

Copy From Previous

input:
  type: copy
  from: artifact-myapp-v1.2.2
  sections: [source, builds.devl]  # What to copy

Promote

input:
  type: promote
  from: artifact-myapp-v1.2.3
  environment: preprod
  source_environment: devl

The input stage IS the mapping from previous → current context.

Repeatable Builds

If the context record captures:

Source commit (exact code)
Build executor version (exact tooling)
All dependency versions
Full input context

Then replay = same outputs. Binary reproducible builds.

replay:
  artifact: artifact-myapp-v1.2.3
  environment: devl
  # System reconstructs exact context, runs same fences

Type Checking (Optional)

Nodes can declare:

requires:
  - repository
  - commit
produces:
  - docker-image
  - jar

Validation traces upstream: does the required type exist in ancestry? Don't care which node produced it, just that it's there.

IDE shows: green arrow = valid connection, red arrow = missing requirement.

The Simplicity

A pipeline stage is:

Mapping YAML (wire context to pattern)
Include pattern (reusable fences)

That's it. Everything else is:

Graph structure (edges with conditions)
Context accumulation (DynamoDB-backed)
Interpolation at runtime

No code generation. No scattered stores. Just graph traversal with accumulating context.

Promotion Flow

Promoting devl → preprod is just running the same pipeline with different conditions:

The Copier Stage

First stage of any pipeline is the input/copier. For promotion:

input:
  type: promote
  find:
    system: myapp
    environment: myenv  
    sub_environment: devl
    # Finds: artifact tagged with these + "promotable" or latest successful
  copy:
    artifact: ${found.artifact_id}
    sections: [source, builds.devl]

The copier:

Queries for artifact matching system/env/sub-env metadata
Finds the one tagged environment:devl (or whatever you're promoting from)
Copies its context as starting point
Pipeline continues with that context

Same Pipeline, Different Paths

Instead of separate pipelines for devl vs prod, use conditional edges:

[Input/Copier]
    ↓ south
[Build Stage]
    ↓ south (condition: needs_build=true)     ↓ south (condition: needs_build=false)
[Gradle Build]                                [Skip - artifact exists]
    ↓ south                                       ↓ south
[Deploy Stage] ←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←┘
    ↓ south (condition: env=devl)    ↓ south (condition: env=prod)
[Deploy to DevL]                     [Deploy to Prod]
    ↓                                    ↓
[End]                                [End]

When promoting:

needs_build=false → skips build, uses existing artifact
Same pipeline definition, conditions select the path
Like two when clauses, but expressed as graph edges

It's Just a Git Commit Lookup

Conceptually, promotion = "find the thing, extract its data, carry on"

promote:
  # Find artifact by metadata
  query:
    system: ${pipeline.system}
    environment: ${pipeline.environment}
    sub_environment: ${promote_from}
    status: successful
    
  # Extract what we need
  extract:
    artifact_id: ${result.artifact_id}
    docker_image: ${result.builds.${promote_from}.outputs.docker}
    source_commit: ${result.source.commit}
    
  # That becomes our starting context
  context:
    promoted_from: ${promote_from}
    artifact: ${extracted}

It's not exactly a git commit lookup, but people will get that analogy.

Type System with Subtypes

Types are colon-separated for hierarchy and filtering:

produces:
  - artifact:jar
  - artifact:docker
  - resource:cloudformation:stack
  - resource:ecs:service

requires:
  - artifact:*           # Any artifact
  - artifact:jar         # Specifically a jar
  - artifact:docker:*    # Any docker artifact
  - resource:*           # Any resource

Subtype Matching

Pattern	Matches
`artifact:*`	artifact:jar, artifact:docker, artifact:war
`artifact:docker`	artifact:docker only
`artifact:docker:*`	artifact:docker:arm64, artifact:docker:amd64
`resource:ecs:*`	resource:ecs:service, resource:ecs:task-definition
`*`	anything

In Conditions

conditions:
  all:
    - type: hasArtifact
      args:
        type: artifact:docker:*
    - type: hasResource
      args:
        type: resource:cloudformation:*

In Mappings

mapping:
  # Find any docker artifact
  docker_image: ${context.artifacts[type=artifact:docker:*].first.uri}
  
  # Find specific jar
  app_jar: ${context.artifacts[type=artifact:jar].first.path}

Peregrine Fundamentals (artifact = function application)
conditions.groovy (condition resolver pattern)
pqis-lb-live-data-pattern (include + context injection example)