template-engine-copier skill: initial import
Copier + Jinja template-repository design skill: reference architecture, implementation workflow, validation checklist, and a full worked example template (questionnaire, extensions, resolvers, wrapper-first .scripts). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,120 @@
|
||||
---
|
||||
name: template-engine-copier
|
||||
description: Design and implement a Copier + Jinja based template repository, following the proven patterns used in iac-tmpl-aws-projects and iac-tmpl-aws-organizations.
|
||||
---
|
||||
|
||||
# template-engine-copier
|
||||
|
||||
Use this skill when you need to **create, refactor, or standardize** a repository that acts as a **Copier template engine** for downstream projects.
|
||||
|
||||
This skill is based on real patterns from:
|
||||
|
||||
- `iac-tmpl-aws-projects/`
|
||||
- `iac-tmpl-aws-organizations/`
|
||||
|
||||
and is meant to guide creation of repositories that have:
|
||||
|
||||
1. clear questionnaire-driven inputs,
|
||||
2. deterministic Jinja rendering,
|
||||
3. conditional folder/file generation,
|
||||
4. safe bootstrapping and re-render workflows.
|
||||
|
||||
It now also includes a **concrete scaffold** you can copy from:
|
||||
|
||||
- `example/`
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Deliver a template repository where operators can run one command like:
|
||||
|
||||
```bash
|
||||
task copier:bootstrap-one -- --copier-target-dir "../target-repo" --copier-params-file "./copier-answers.yaml"
|
||||
```
|
||||
|
||||
and get a rendered repository with predictable structure and preserved answer history.
|
||||
|
||||
---
|
||||
|
||||
## Read in this order
|
||||
|
||||
1. `instructions/01-reference-architecture.md`
|
||||
2. `instructions/02-implementation-workflow.md`
|
||||
3. `instructions/03-validation-checklist.md`
|
||||
4. `example/README.md` and `example/`
|
||||
|
||||
---
|
||||
|
||||
## Canonical architecture (from analyzed repos)
|
||||
|
||||
```text
|
||||
<template-repo>/
|
||||
.copier/
|
||||
copier.yaml
|
||||
questionnaire/*.yml
|
||||
extensions/
|
||||
context_hook_extension/ # optional but recommended for complex templates
|
||||
resolvers/
|
||||
.scripts/
|
||||
copier/
|
||||
Taskfile.yml
|
||||
api/bootstrap-one.sh
|
||||
lib/
|
||||
--env-vars-reader.sh
|
||||
--index.sh
|
||||
--index-api.sh
|
||||
-bootstrap-one.sh
|
||||
answers/
|
||||
--index-api.sh
|
||||
-new-ensure-existance.sh
|
||||
-previous-pull-and-merge.sh
|
||||
-params-pull-and-merge.sh
|
||||
base/
|
||||
loggers/
|
||||
Taskfile.yml
|
||||
{{ _copier_conf.answers_file }}.jinja
|
||||
{{ '.tmp-gitlab-ci.yml' }}.jinja
|
||||
src/**/*.jinja
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Non-negotiable conventions
|
||||
|
||||
1. Keep Copier source-of-truth in `.copier/copier.yaml`.
|
||||
2. Keep questions separated under `.copier/questionnaire/*.yml` and include via `!include`.
|
||||
3. Use `.jinja` suffix and render both file contents and file/folder names.
|
||||
4. For dynamic instance folders, support `[INDEX]` expansion pattern via context hook.
|
||||
5. Keep operational entrypoint wrapper-first: `Taskfile -> api -> lib`.
|
||||
6. Merge answers in this order:
|
||||
- previous target answers,
|
||||
- explicit params file,
|
||||
- final merged file used for `copier copy --data-file ...`.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns to avoid
|
||||
|
||||
1. Putting all logic directly into one giant Jinja file.
|
||||
2. Copying templates without answer merge strategy (breaks update flows).
|
||||
3. Embedding one-off shell logic directly in Taskfile instead of API/lib wrappers.
|
||||
4. Using ad-hoc variable names instead of stable normalized `flat.*` style for complex conditions.
|
||||
5. Ignoring render-time safety for generated CI files (e.g., temp naming + controlled rename).
|
||||
|
||||
---
|
||||
|
||||
## What to copy first (practical)
|
||||
|
||||
When building a new template repo, copy these parts first from `example/`:
|
||||
|
||||
1. `.copier/copier.yaml`
|
||||
2. `.copier/questionnaire/0100-template.yml`
|
||||
3. `.copier/extensions/context_hook_extension/hook.py`
|
||||
4. `.copier/extensions/resolvers/resolve_safe_identifier.py`
|
||||
5. `.scripts/copier/` (all files)
|
||||
6. `Taskfile.yml`
|
||||
7. `{{ _copier_conf.answers_file }}.jinja`
|
||||
8. `{{ '.tmp-gitlab-ci.yml' }}.jinja`
|
||||
|
||||
Then adapt `src/**/*.jinja` for your domain-specific resources.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
_template_suffix: ".jinja"
|
||||
|
||||
_exclude:
|
||||
- ".git"
|
||||
- ".copier"
|
||||
- ".scripts"
|
||||
- ".venv"
|
||||
- "Taskfile.yml"
|
||||
- "pyproject.toml"
|
||||
- "copier-answers-new.yml"
|
||||
|
||||
_jinja_extensions:
|
||||
- copier_templates_extensions.TemplateExtensionLoader
|
||||
- .copier/extensions/context_hook_extension/hook.py:ContextHookExtension
|
||||
- .copier/extensions/resolvers/resolve_safe_identifier.py:SafeIdentifierExtension
|
||||
|
||||
_answers_file: copier-answers.yml
|
||||
|
||||
!include .copier/questionnaire/*.yml
|
||||
@@ -0,0 +1,72 @@
|
||||
import os
|
||||
import shutil
|
||||
|
||||
from copier_templates_extensions import ContextHook
|
||||
|
||||
|
||||
class ContextHookExtension(ContextHook):
|
||||
update = False
|
||||
|
||||
def hook(self, context):
|
||||
_ensure_flat_variables(context)
|
||||
_ensure_resources_instances(context)
|
||||
return context
|
||||
|
||||
|
||||
def _ensure_flat_variables(context):
|
||||
if 'flat' not in context or not isinstance(context.get('flat'), dict):
|
||||
context['flat'] = {}
|
||||
|
||||
metadata = context.get('template_metadata', {}) if isinstance(context, dict) else {}
|
||||
features = context.get('features', {}) if isinstance(context, dict) else {}
|
||||
modules = features.get('modules', {}) if isinstance(features, dict) else {}
|
||||
|
||||
domain_name = metadata.get('domain', '') if isinstance(metadata, dict) else ''
|
||||
project_name = metadata.get('project', '') if isinstance(metadata, dict) else ''
|
||||
|
||||
context['flat']['domain_name'] = domain_name
|
||||
context['flat']['project_name'] = project_name
|
||||
|
||||
instances = modules.get('instances', []) if isinstance(modules, dict) else []
|
||||
is_enabled = bool(modules.get('is_enabled', False)) if isinstance(modules, dict) else False
|
||||
|
||||
names = []
|
||||
for instance in instances if isinstance(instances, list) else []:
|
||||
if isinstance(instance, dict) and instance.get('name'):
|
||||
names.append(f"{domain_name}-{project_name}-{instance['name']}")
|
||||
|
||||
context['flat']['modules_is_enabled'] = is_enabled
|
||||
context['flat']['module_instance_names'] = names
|
||||
|
||||
|
||||
def _ensure_resources_instances(context):
|
||||
"""Expand folders containing [INDEX] into concrete per-instance directories."""
|
||||
base_path = './src'
|
||||
if not os.path.exists(base_path):
|
||||
return
|
||||
|
||||
targets = []
|
||||
for root, dirs, _ in os.walk(base_path):
|
||||
for d in dirs:
|
||||
if '[INDEX]' in d:
|
||||
targets.append(os.path.join(root, d))
|
||||
|
||||
for path in targets:
|
||||
parent = os.path.dirname(path)
|
||||
instances = context.get('flat', {}).get('module_instance_names', [])
|
||||
for idx, name in enumerate(instances):
|
||||
dst = os.path.join(parent, name)
|
||||
if os.path.exists(dst):
|
||||
continue
|
||||
os.makedirs(dst, exist_ok=True)
|
||||
for fn in os.listdir(path):
|
||||
src_file = os.path.join(path, fn)
|
||||
if not os.path.isfile(src_file):
|
||||
continue
|
||||
with open(src_file, 'r', encoding='utf-8') as f:
|
||||
content = f.read().replace('INDEX', str(idx))
|
||||
with open(os.path.join(dst, fn), 'w', encoding='utf-8') as f:
|
||||
f.write(content)
|
||||
|
||||
if os.path.exists(path):
|
||||
shutil.rmtree(path)
|
||||
@@ -0,0 +1,17 @@
|
||||
from jinja2.ext import Extension
|
||||
|
||||
|
||||
def resolve_safe_identifier(name: str) -> str:
|
||||
return (
|
||||
name
|
||||
.replace("-", "_")
|
||||
.replace("{", "")
|
||||
.replace("}", "")
|
||||
.replace(".", "_")
|
||||
)
|
||||
|
||||
|
||||
class SafeIdentifierExtension(Extension):
|
||||
def __init__(self, environment):
|
||||
super().__init__(environment)
|
||||
environment.globals["resolve_safe_identifier"] = resolve_safe_identifier
|
||||
@@ -0,0 +1 @@
|
||||
from .loggers import info, warning, debug
|
||||
@@ -0,0 +1,16 @@
|
||||
import logging
|
||||
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
|
||||
|
||||
def info(name, message, extra=None):
|
||||
logging.getLogger(name).info(message)
|
||||
|
||||
|
||||
def warning(name, message, extra=None):
|
||||
logging.getLogger(name).warning(message)
|
||||
|
||||
|
||||
def debug(name, message, extra=None):
|
||||
logging.getLogger(name).debug(message)
|
||||
@@ -0,0 +1,18 @@
|
||||
template_metadata:
|
||||
type: yaml
|
||||
help: "Generic template metadata used by naming and rendering hooks"
|
||||
default: |
|
||||
domain: demo
|
||||
project: sample-product
|
||||
|
||||
features:
|
||||
type: yaml
|
||||
help: "Feature/module matrix for template rendering"
|
||||
default: |
|
||||
modules:
|
||||
is_enabled: true
|
||||
instances:
|
||||
- name: users
|
||||
version: 1.0.0
|
||||
- name: billing
|
||||
version: 1.0.0
|
||||
@@ -0,0 +1,9 @@
|
||||
version: "3"
|
||||
|
||||
tasks:
|
||||
bootstrap-one:
|
||||
desc: -- --copier-force-copy "<true|false>" --copier-target-dir "<path>" --copier-params-file "<path>"
|
||||
cmds:
|
||||
- |
|
||||
./.scripts/copier/api/bootstrap-one.sh {{ .CLI_ARGS }}
|
||||
silent: true
|
||||
@@ -0,0 +1,7 @@
|
||||
#!/bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
. ./.scripts/copier/lib/--index.sh "$@"
|
||||
|
||||
_copier_bootstrap_one
|
||||
@@ -0,0 +1,41 @@
|
||||
#!/bin/bash
|
||||
|
||||
export HL_V1_COPIER_TARGET_DIR=""
|
||||
export HL_V1_COPIER_FORCE_COPY=false
|
||||
export HL_V1_COPIER_PARAMS_FILE=""
|
||||
export HL_V1_COPIER_FILE_COPIER_ANSWERS="copier-answers.yml"
|
||||
export HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW="copier-answers-new.yml"
|
||||
|
||||
ALL_ARGS=("$@")
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--copier-target-dir=*)
|
||||
export HL_V1_COPIER_TARGET_DIR="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--copier-target-dir)
|
||||
export HL_V1_COPIER_TARGET_DIR="$2"
|
||||
shift 2
|
||||
;;
|
||||
--copier-force-copy=*)
|
||||
export HL_V1_COPIER_FORCE_COPY="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--copier-force-copy)
|
||||
export HL_V1_COPIER_FORCE_COPY="$2"
|
||||
shift 2
|
||||
;;
|
||||
--copier-params-file=*)
|
||||
export HL_V1_COPIER_PARAMS_FILE="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--copier-params-file)
|
||||
export HL_V1_COPIER_PARAMS_FILE="$2"
|
||||
shift 2
|
||||
;;
|
||||
*)
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
set -- "${ALL_ARGS[@]}"
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/bin/bash
|
||||
|
||||
. ./.scripts/copier/lib/answers/--index-api.sh
|
||||
. ./.scripts/copier/lib/-bootstrap-one.sh
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/bin/bash
|
||||
|
||||
. ./.scripts/copier/lib/--env-vars-reader.sh "$@"
|
||||
. ./.scripts/copier/lib/--index-api.sh
|
||||
@@ -0,0 +1,33 @@
|
||||
#!/bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
_copier_bootstrap_one() {
|
||||
[ -z "${HL_V1_COPIER_TARGET_DIR}" ] && {
|
||||
echo "--copier-target-dir is required"
|
||||
return 1
|
||||
}
|
||||
|
||||
_copier_answers_new_ensure_existence
|
||||
_copier_answers_previous_pull_and_merge
|
||||
_copier_answers_params_pull_and_merge
|
||||
|
||||
if [ "${HL_V1_COPIER_FORCE_COPY}" = "true" ]; then
|
||||
copier copy \
|
||||
--trust \
|
||||
--force \
|
||||
--overwrite \
|
||||
--data-file "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}" \
|
||||
. "${HL_V1_COPIER_TARGET_DIR}"
|
||||
else
|
||||
copier copy \
|
||||
--trust \
|
||||
--data-file "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}" \
|
||||
. "${HL_V1_COPIER_TARGET_DIR}"
|
||||
fi
|
||||
|
||||
# Keep template repo CI separate from rendered repo CI, same pattern as source repos.
|
||||
if [ -f "${HL_V1_COPIER_TARGET_DIR}/.tmp-gitlab-ci.yml" ]; then
|
||||
mv "${HL_V1_COPIER_TARGET_DIR}/.tmp-gitlab-ci.yml" "${HL_V1_COPIER_TARGET_DIR}/.gitlab-ci.yml"
|
||||
fi
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
#!/bin/bash
|
||||
|
||||
. ./.scripts/copier/lib/answers/-new-ensure-existance.sh
|
||||
. ./.scripts/copier/lib/answers/-previous-pull-and-merge.sh
|
||||
. ./.scripts/copier/lib/answers/-params-pull-and-merge.sh
|
||||
@@ -0,0 +1,5 @@
|
||||
#!/bin/bash
|
||||
|
||||
_copier_answers_new_ensure_existence() {
|
||||
[ -f "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}" ] || touch "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}"
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
#!/bin/bash
|
||||
|
||||
_copier_answers_params_pull_and_merge() {
|
||||
[ -f "${HL_V1_COPIER_PARAMS_FILE}" ] || return 0
|
||||
[ -f "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}" ] || return 0
|
||||
|
||||
if command -v yq >/dev/null 2>&1; then
|
||||
local TMP_WORK_FILE="tmp-work-file.yml"
|
||||
yq eval-all '. as $item ireduce ({}; . * $item)' \
|
||||
"${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}" \
|
||||
"${HL_V1_COPIER_PARAMS_FILE}" >"${TMP_WORK_FILE}"
|
||||
mv "${TMP_WORK_FILE}" "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}"
|
||||
else
|
||||
# Fallback: params file wins as full override when yq is unavailable.
|
||||
cp "${HL_V1_COPIER_PARAMS_FILE}" "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}"
|
||||
fi
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
#!/bin/bash
|
||||
|
||||
_copier_answers_previous_pull_and_merge() {
|
||||
local PREVIOUS="${HL_V1_COPIER_TARGET_DIR}/${HL_V1_COPIER_FILE_COPIER_ANSWERS}"
|
||||
[ -f "${PREVIOUS}" ] && cp "${PREVIOUS}" "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}" || true
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
# Example Template Repository
|
||||
|
||||
This is a minimal but concrete Copier + Jinja template repository scaffold.
|
||||
|
||||
It is intentionally **domain-generic** and demonstrates:
|
||||
|
||||
- metadata + features questionnaire model,
|
||||
- dynamic module/component folder generation,
|
||||
- safe identifier resolver,
|
||||
- shell-based bootstrap/answers merge flow.
|
||||
|
||||
## Render example
|
||||
|
||||
```bash
|
||||
task copier:bootstrap-one -- --copier-target-dir "../rendered-example" --copier-params-file "./copier-answers.example.yaml"
|
||||
```
|
||||
@@ -0,0 +1,4 @@
|
||||
version: "3"
|
||||
|
||||
includes:
|
||||
copier: ./.scripts/copier/Taskfile.yml
|
||||
@@ -0,0 +1,12 @@
|
||||
template_metadata:
|
||||
domain: demo
|
||||
project: sample-product
|
||||
|
||||
features:
|
||||
modules:
|
||||
is_enabled: true
|
||||
instances:
|
||||
- name: users
|
||||
version: 1.0.0
|
||||
- name: billing
|
||||
version: 1.0.0
|
||||
@@ -0,0 +1,8 @@
|
||||
[project]
|
||||
name = "example-template-repo"
|
||||
version = "0.1.0"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"copier==9.14.0",
|
||||
"copier-templates-extensions==0.3.2",
|
||||
]
|
||||
@@ -0,0 +1,21 @@
|
||||
{% macro render() %}
|
||||
{% set metadata = template_metadata if template_metadata is mapping else {} %}
|
||||
{% set features_map = features if features is mapping else {} %}
|
||||
{% set modules = features_map.get('modules', {}) if features_map else {} %}
|
||||
|
||||
Template Domain: {{ metadata.get('domain', 'unknown-domain') }}
|
||||
Template Project: {{ metadata.get('project', 'unknown-project') }}
|
||||
|
||||
{% if modules.get('is_enabled', false) -%}
|
||||
Include module components from ./components/
|
||||
{% else -%}
|
||||
No modules enabled.
|
||||
{% endif %}
|
||||
{% endmacro %}
|
||||
|
||||
{%- set result = render() -%}
|
||||
{%- for line in result.split('\n') -%}
|
||||
{% if line | trim | length != 0 -%}
|
||||
{{ line }}
|
||||
{% endif %}
|
||||
{%- endfor %}
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
{% set instances_names = flat.get('module_instance_names', []) %}
|
||||
|
||||
{% for item in instances_names %}
|
||||
component: {{ resolve_safe_identifier(item) }}
|
||||
{% endfor %}
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
component_id={{ resolve_safe_identifier(flat.module_instance_names[INDEX]) }}
|
||||
component_name={{ flat.module_instance_names[INDEX] }}
|
||||
generated_with=copier+jinja
|
||||
@@ -0,0 +1,17 @@
|
||||
stages:
|
||||
- lint
|
||||
- test
|
||||
|
||||
template-content-check:
|
||||
stage: lint
|
||||
image: alpine:3.20
|
||||
script:
|
||||
- test -f README.md
|
||||
- test -f copier-answers.example.yaml
|
||||
- test -d src
|
||||
|
||||
template-smoke-test:
|
||||
stage: test
|
||||
image: alpine:3.20
|
||||
script:
|
||||
- echo "Template repository scaffold is present"
|
||||
@@ -0,0 +1,2 @@
|
||||
# Changes here will be overwritten by Copier; NEVER EDIT MANUALLY
|
||||
{{ _copier_answers|to_nice_yaml -}}
|
||||
@@ -0,0 +1,218 @@
|
||||
# Copier + Jinja Template Engine Reference Architecture
|
||||
|
||||
This reference architecture is synthesized from:
|
||||
|
||||
- `iac-tmpl-aws-projects/`
|
||||
- `iac-tmpl-aws-organizations/`
|
||||
|
||||
and should be treated as the default blueprint for new template repositories.
|
||||
|
||||
## 1) Copier core contract
|
||||
|
||||
Use `.copier/copier.yaml` as central control plane.
|
||||
|
||||
### Recommended shape
|
||||
|
||||
```yaml
|
||||
_template_suffix: ".jinja"
|
||||
|
||||
_exclude:
|
||||
- ".git"
|
||||
- ".copier"
|
||||
- ".scripts"
|
||||
- ".venv"
|
||||
- ".devbox"
|
||||
- "devbox.json"
|
||||
- "devbox.lock"
|
||||
- ".gitlab-ci.yml"
|
||||
- "Taskfile.yml"
|
||||
- "pyproject.toml"
|
||||
- "uv.lock"
|
||||
- "copier-answers-new.yml"
|
||||
- "copier-answers.example.yaml"
|
||||
|
||||
_jinja_extensions:
|
||||
- copier_templates_extensions.TemplateExtensionLoader
|
||||
# plus custom extension(s)
|
||||
|
||||
_answers_file: copier-answers.yml
|
||||
|
||||
!include .copier/questionnaire/*.yml
|
||||
```
|
||||
|
||||
## 2) Questionnaire strategy
|
||||
|
||||
Keep questions modular under `.copier/questionnaire/`.
|
||||
|
||||
Why:
|
||||
|
||||
- supports domain separation,
|
||||
- scales better than one monolithic YAML,
|
||||
- makes review and versioning easier.
|
||||
|
||||
## 3) Context shaping strategy
|
||||
|
||||
For advanced templates (especially conditional directories + repeated resource modules), use custom context hook extension.
|
||||
|
||||
Pattern used in `iac-tmpl-aws-projects`:
|
||||
|
||||
- `.copier/extensions/context_hook_extension/hook.py`
|
||||
- merged orchestration in `hook_function.py`
|
||||
1. compute/normalize flat variables,
|
||||
2. expand `[INDEX]` instance folders.
|
||||
|
||||
### Why `flat.*` variables
|
||||
|
||||
They reduce deep nested path usage in Jinja and make conditional file/folder naming concise:
|
||||
|
||||
- `flat.dev_is_enabled`
|
||||
- `flat.dev_vm_is_enabled`
|
||||
- `flat.dev_vm_instances_names`
|
||||
|
||||
They should also carry any normalized/safe identifiers needed by templates:
|
||||
|
||||
- `flat.project_safe`
|
||||
- `flat.environment_safe`
|
||||
- `flat.component_safe_names`
|
||||
|
||||
Rule: do **not** repeat normalization logic such as `lower | replace(...) | trim` throughout template paths or file bodies. If a safe identifier is needed more than once, calculate it once in the context hook (or a deterministic resolver) and expose it as a stable context variable like `flat.project_safe`.
|
||||
|
||||
## 4) Dynamic folder generation (`[INDEX]`)
|
||||
|
||||
In `iac-tmpl-aws-projects`, folders like:
|
||||
|
||||
```text
|
||||
iac/{{ 'dev' if flat.dev_is_enabled }}/.../{{ flat.dev_vm_instances_names[INDEX] }}/
|
||||
```
|
||||
|
||||
are expanded by context hook functions that:
|
||||
|
||||
1. find `[INDEX]` patterns,
|
||||
2. map pattern -> context list,
|
||||
3. create concrete per-instance folders,
|
||||
4. replace `INDEX` placeholders in contents,
|
||||
5. remove the template placeholder folder.
|
||||
|
||||
## 5) Custom Jinja resolvers
|
||||
|
||||
Keep custom Jinja helpers in `.copier/extensions/resolvers/` and register them in `_jinja_extensions`.
|
||||
|
||||
Examples discovered:
|
||||
|
||||
- `resolve_safe_tf_module_name(...)`
|
||||
- `resolve_regions(...)`
|
||||
- `resolve_availability_zones(...)`
|
||||
- tags resolver(s) for project/organization context.
|
||||
|
||||
Rule: put transformation logic in extensions, not inline repeated Jinja blocks.
|
||||
|
||||
For safe names specifically, prefer context-hook derived variables over inline filter chains. Templates should read:
|
||||
|
||||
```text
|
||||
applications/{{ flat.project_safe }}/...
|
||||
```
|
||||
|
||||
not:
|
||||
|
||||
```text
|
||||
applications/{{ project.name | lower | replace(' ', '-') | replace('_', '-') }}/...
|
||||
```
|
||||
|
||||
## 6) Operational wrapper architecture
|
||||
|
||||
Use shell wrapper-first command surface:
|
||||
|
||||
`Taskfile -> api -> lib`
|
||||
|
||||
Concrete layout:
|
||||
|
||||
- `.scripts/copier/Taskfile.yml` - operator tasks
|
||||
- `.scripts/copier/api/bootstrap-one.sh` - thin wrapper
|
||||
- `.scripts/copier/lib/*.sh` - actual behavior
|
||||
|
||||
This keeps CLI UX stable while allowing internal evolution.
|
||||
|
||||
### Bootstrap from a temporary render workspace
|
||||
|
||||
Follow the `iac-tmpl-aws-projects` runtime pattern: the public API wrapper should not render directly from the live template repo folder. It should first create a sibling temporary copy and execute rendering from there.
|
||||
|
||||
Canonical flow:
|
||||
|
||||
1. Capture the live template folder name from `$PWD`.
|
||||
2. Remove any previous sibling temp folder, e.g. `../tmp-${TEMPLATE_FOLDER}`.
|
||||
3. Copy the live template folder to `../tmp-${TEMPLATE_FOLDER}`.
|
||||
4. `cd` into the temp folder.
|
||||
5. Load logger and copier indexes from the temp folder.
|
||||
6. Run `_copier_bootstrap_one` from the temp folder.
|
||||
7. Return to the live template folder after rendering.
|
||||
|
||||
Inside `_copier_bootstrap_one`, before invoking `copier copy`:
|
||||
|
||||
1. Copy `.copier/copier.yaml` to root `copier.yaml` when the template stores Copier config under `.copier/`.
|
||||
2. Remove temp-copy internals that interfere with Copier local rendering:
|
||||
- `.git`
|
||||
- `.devbox`
|
||||
- `.venv`
|
||||
3. Then run `copier copy --trust --data-file ... . "$TARGET_DIR"` from the temp folder.
|
||||
|
||||
Why this matters:
|
||||
|
||||
- Copier sees a plain local template directory, not the live Git repository.
|
||||
- It avoids Git/tag discovery noise such as `No git tags found in template; using HEAD as ref`.
|
||||
- It prevents the render from mutating or deleting files in the live template repo.
|
||||
- It keeps generated transient files (`copier.yaml`, `copier-answers-new.yml`, temp merge files) isolated in the temp copy.
|
||||
|
||||
Do not solve local bootstrap Git warnings by creating tags just to satisfy Copier. Tags are for intentional template releases, not for local render hygiene.
|
||||
|
||||
## 7) Answers merge pipeline
|
||||
|
||||
Use 3-step pre-render merge flow before `copier copy`:
|
||||
|
||||
1. ensure `copier-answers-new.yml` exists,
|
||||
2. pull previous target answers (`<target>/copier-answers.yml`) if present,
|
||||
3. merge explicit params file into the working answers file.
|
||||
|
||||
Then execute:
|
||||
|
||||
```bash
|
||||
copier copy --trust --data-file "copier-answers-new.yml" ...
|
||||
```
|
||||
|
||||
## 8) Generated CI file safety pattern
|
||||
|
||||
When template repo itself has CI files that must not be copied literally, use a temporary rendered filename:
|
||||
|
||||
- template file: `{{ '.tmp-gitlab-ci.yml' }}.jinja`
|
||||
- post-render rename in bootstrap lib:
|
||||
- `.tmp-gitlab-ci.yml` -> `.gitlab-ci.yml`
|
||||
|
||||
## 9) Dependency baseline
|
||||
|
||||
Keep Copier stack pinned in `pyproject.toml`:
|
||||
|
||||
- `copier`
|
||||
- `copier-templates-extensions`
|
||||
- supportive tooling (e.g. `check-jsonschema`, logging libs)
|
||||
|
||||
Use lockfile (`uv.lock`) for deterministic local behavior.
|
||||
|
||||
## 10) Where this is implemented in this skill
|
||||
|
||||
See concrete scaffold:
|
||||
|
||||
- `example/.copier/copier.yaml`
|
||||
- `example/.copier/questionnaire/0100-template.yml`
|
||||
- `example/.copier/extensions/context_hook_extension/hook.py`
|
||||
- `example/.copier/extensions/resolvers/resolve_safe_identifier.py`
|
||||
- `example/.scripts/copier/`
|
||||
|
||||
|
||||
## 11) Environment variable prefix convention
|
||||
|
||||
Before implementing shell wrappers, **propose and confirm** the environment variable prefix with the user (e.g. `HL_V1_`, `PALANTIR_`, etc.).
|
||||
|
||||
Rules:
|
||||
|
||||
- Do not hardcode an inherited prefix from old scaffolds without confirmation.
|
||||
- Keep one prefix consistently across `--env-vars-reader.sh`, bootstrap libs, and answer merge libs.
|
||||
- Document the chosen prefix in README or implementation notes.
|
||||
@@ -0,0 +1,156 @@
|
||||
# Implementation Workflow (Copier + Jinja Template Repository)
|
||||
|
||||
Follow this sequence when creating a new template-engine repository.
|
||||
|
||||
## Phase 1 — Define rendering contract
|
||||
|
||||
1. Define template scope (what downstream repository you generate).
|
||||
2. Identify stable input schema sections (e.g. tenant/project/environments/resources).
|
||||
3. Decide what is:
|
||||
- static files,
|
||||
- conditional files/folders,
|
||||
- repeatable per-instance blocks.
|
||||
|
||||
Deliverable: first draft of `copier-answers.example.yaml`.
|
||||
|
||||
## Phase 2 — Bootstrap Copier skeleton
|
||||
|
||||
Create:
|
||||
|
||||
- `.copier/copier.yaml`
|
||||
- `.copier/questionnaire/*.yml`
|
||||
- `{{ _copier_conf.answers_file }}.jinja`
|
||||
|
||||
Use baseline:
|
||||
|
||||
- `_template_suffix: ".jinja"`
|
||||
- modular `!include .copier/questionnaire/*.yml`
|
||||
- explicit `_exclude` list
|
||||
- `_answers_file: copier-answers.yml`
|
||||
|
||||
## Phase 3 — Create operational wrappers
|
||||
|
||||
Implement wrapper chain:
|
||||
|
||||
1. root `Taskfile.yml` includes `.scripts/copier/Taskfile.yml`
|
||||
2. `.scripts/copier/Taskfile.yml` exposes `bootstrap-one`
|
||||
3. `.scripts/copier/api/bootstrap-one.sh` remains thin
|
||||
4. **confirm ENV var prefix with user** (or propose one) before implementing lib internals
|
||||
5. `.scripts/copier/lib/` contains logic and answers merge helpers
|
||||
|
||||
Minimum behavior in lib:
|
||||
|
||||
- parse CLI args with both styles: `--arg value` and `--arg=value` (for `--copier-target-dir`, `--copier-force-copy`, `--copier-params-file`),
|
||||
- prepare merged answers file,
|
||||
- run `copier copy --trust --data-file ...` from a sibling temporary copy of the template repo, not from the live template repo folder,
|
||||
- post-render sanitization/renames if needed.
|
||||
|
||||
Mandatory temp-copy bootstrap pattern from `iac-tmpl-aws-projects`:
|
||||
|
||||
- In `api/bootstrap-one.sh`:
|
||||
1. derive the live template folder name from `basename "$PWD"`,
|
||||
2. delete any old sibling `../tmp-${TEMPLATE_FOLDER}`,
|
||||
3. copy the whole template repo into `../tmp-${TEMPLATE_FOLDER}`,
|
||||
4. `cd` into the temp folder,
|
||||
5. source `.scripts/loggers/lib/--index.sh` and `.scripts/copier/lib/--index.sh`,
|
||||
6. execute `_copier_bootstrap_one`,
|
||||
7. return to the original live template folder.
|
||||
- In `lib/-bootstrap-one.sh`, before `copier copy`:
|
||||
1. copy `.copier/copier.yaml` to root `copier.yaml` if the repo uses `.copier/copier.yaml`,
|
||||
2. remove `.git`, `.devbox`, and `.venv` from the temp copy,
|
||||
3. run Copier from the temp folder into the real target.
|
||||
|
||||
Do **not** create Git tags merely to silence Copier's local template version warning. The preferred local bootstrap hygiene is rendering from the sanitized temp copy. Use `--vcs-ref=HEAD` only when there is an explicit reason to render directly from a Git-backed live folder.
|
||||
|
||||
## Phase 4 — Add Jinja template layers
|
||||
|
||||
1. Add top-level templates (`iac/main.tf.jinja`, providers, backend, etc.).
|
||||
2. Gate directories via folder-name expressions:
|
||||
- `{{ 'dev' if flat.dev_is_enabled }}`
|
||||
3. Gate module calls inside files via `{% if ... %}`.
|
||||
|
||||
Pattern:
|
||||
|
||||
- folder condition controls existence,
|
||||
- inner condition controls file content clarity.
|
||||
|
||||
**Mandatory wrapped render-path rule:**
|
||||
|
||||
- For every optional module/resource, apply conditions at the **path level** (folder/file names), not only inside file contents.
|
||||
- Preferred pattern (from `iac-tmpl-aws-projects`):
|
||||
- `iac/{{ 'dev' if flat.dev_is_enabled }}/{{ 'compute' if flat.dev_compute_is_enabled }}/main.tf.jinja`
|
||||
- Apply this systematically to all optional Jinja outputs so disabled features do not produce placeholder files/directories.
|
||||
|
||||
**Mandatory safe-variable rule:**
|
||||
|
||||
- Do not embed repeated normalization/sanitization logic in template paths or file bodies, e.g. avoid:
|
||||
- `{{ project.name | lower | replace(' ', '-') | replace('_', '-') | replace('.', '-') }}`
|
||||
- If a safe name is needed, calculate it once in the context hook and expose a named variable, e.g.:
|
||||
- `flat.project_safe`
|
||||
- `flat.environment_safe`
|
||||
- `flat.component_safe_names`
|
||||
- Use the named variable in templates:
|
||||
- `applications/{{ flat.project_safe }}/main.yaml.jinja`
|
||||
- If a one-off transformation is genuinely needed, prefer a small deterministic resolver over duplicating filter chains.
|
||||
|
||||
## Phase 5 — Add custom extensions (if complexity requires)
|
||||
|
||||
When logic gets repetitive, move it into extensions.
|
||||
|
||||
### Resolver extension examples
|
||||
|
||||
- normalize names: `resolve_safe_tf_module_name(name)`
|
||||
- derive tags / regions / zones
|
||||
|
||||
### Context hook extension examples
|
||||
|
||||
- calculate `flat.*` shortcuts,
|
||||
- calculate safe/normalized names such as `flat.project_safe` and `flat.environment_safe`,
|
||||
- expand `[INDEX]` directories into per-instance folders.
|
||||
|
||||
Register in `.copier/copier.yaml` under `_jinja_extensions`.
|
||||
|
||||
## Phase 6 — Implement answers merge strategy
|
||||
|
||||
Use this exact order to avoid destructive updates:
|
||||
|
||||
1. `copier-answers-new.yml` create if missing,
|
||||
2. copy `<target>/copier-answers.yml` if present,
|
||||
3. merge params file into working answers file (`yq eval-all ...`).
|
||||
|
||||
Then run copier using merged file.
|
||||
|
||||
## Phase 7 — Handle template/self-repo conflicts
|
||||
|
||||
If template repo contains files that should not be copied 1:1 (e.g. self CI):
|
||||
|
||||
1. render as temp name (`.tmp-gitlab-ci.yml`),
|
||||
2. rename in target during bootstrap.
|
||||
|
||||
## Phase 8 — Add docs and examples
|
||||
|
||||
Add/update:
|
||||
|
||||
- `README.md` with local setup and bootstrap command,
|
||||
- `copier-answers.example.yaml` with realistic schema examples,
|
||||
- optional `bootstrap-one-test` task for local maintainer testing.
|
||||
|
||||
## Phase 9 — Validate end-to-end
|
||||
|
||||
Run two critical scenarios:
|
||||
|
||||
1. fresh render into empty target,
|
||||
2. re-render into existing target with existing `copier-answers.yml`.
|
||||
|
||||
Both must succeed without manual edits.
|
||||
|
||||
## Quick-start from this skill example
|
||||
|
||||
If you want to accelerate implementation, copy `example/` and then:
|
||||
|
||||
1. expand questionnaire files,
|
||||
2. expand `flat.*` derivation logic in context hook,
|
||||
3. add domain-specific resolver extensions,
|
||||
4. extend `iac/**/*.jinja` resource tree,
|
||||
5. keep bootstrap/answers flow unchanged unless you have a strong reason.
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
# Validation Checklist (Template Engine Repository)
|
||||
|
||||
Use this checklist before considering Copier/Jinja template work complete.
|
||||
|
||||
## A. Copier configuration
|
||||
|
||||
- [ ] `.copier/copier.yaml` exists and is valid YAML.
|
||||
- [ ] `_template_suffix` is set to `.jinja`.
|
||||
- [ ] `_answers_file` is explicitly configured.
|
||||
- [ ] `_exclude` blocks repo-internal operational files (`.copier`, `.scripts`, env/lock files as needed).
|
||||
- [ ] Questionnaire is loaded via `!include .copier/questionnaire/*.yml`.
|
||||
|
||||
## B. Input schema and answers
|
||||
|
||||
- [ ] `copier-answers.example.yaml` reflects real expected shape.
|
||||
- [ ] Required keys used in templates are documented in the example answers.
|
||||
- [ ] `{{ _copier_conf.answers_file }}.jinja` renders `_copier_answers|to_nice_yaml`.
|
||||
|
||||
## C. Wrapper runtime surface
|
||||
|
||||
- [ ] Root `Taskfile.yml` includes copier module Taskfile.
|
||||
- [ ] `.scripts/copier/Taskfile.yml` has an operator-friendly `bootstrap-one` task.
|
||||
- [ ] API wrapper (`api/bootstrap-one.sh`) creates and enters a sibling `tmp-<template-folder>` copy before sourcing libs and rendering.
|
||||
- [ ] API wrapper returns to the live template repo after the temp-copy render completes.
|
||||
- [ ] ENV variable prefix is explicitly confirmed/proposed and consistently applied across wrapper libs.
|
||||
- [ ] Lib layer parses CLI args for:
|
||||
- [ ] Lib layer parses CLI args for **both** forms: `--arg value` and `--arg=value`.
|
||||
- [ ] `--copier-target-dir`
|
||||
- [ ] `--copier-force-copy`
|
||||
- [ ] `--copier-params-file`
|
||||
|
||||
## D. Answers merge behavior
|
||||
|
||||
- [ ] `copier-answers-new.yml` is created if missing.
|
||||
- [ ] Existing target answers are imported when present.
|
||||
- [ ] Params file is merged into working answers file.
|
||||
- [ ] Render uses `--data-file` pointing to merged answers file.
|
||||
- [ ] Local bootstrap render runs from the sanitized temp copy, not directly from the live Git-backed template folder.
|
||||
- [ ] Temp copy removes `.git`, `.devbox`, and `.venv` before `copier copy` to avoid Copier/Git interference.
|
||||
- [ ] If Copier config lives under `.copier/copier.yaml`, temp-copy bootstrap copies it to root `copier.yaml` before rendering.
|
||||
- [ ] No Git tags are created merely to suppress Copier local render warnings.
|
||||
|
||||
## E. Templating behavior
|
||||
|
||||
- [ ] Conditional folder names render correctly (e.g. `{{ 'dev' if ... }}`).
|
||||
- [ ] **All optional resources** are path-wrapped with Jinja conditions (folder and/or file names), not only guarded in file bodies.
|
||||
- [ ] Conditional Jinja blocks in files match folder-level conditions.
|
||||
- [ ] Safe/normalized names are provided by context-hook or resolver variables (e.g. `flat.project_safe`), not repeated inline filter chains.
|
||||
- [ ] Template paths and file bodies do not duplicate complex expressions like `lower | replace(...) | trim` for identifiers used in more than one place.
|
||||
- [ ] Any generated temporary filenames (e.g. `.tmp-gitlab-ci.yml`) are renamed correctly post-render.
|
||||
|
||||
## F. Extensions (if used)
|
||||
|
||||
- [ ] All custom extensions are registered in `_jinja_extensions`.
|
||||
- [ ] Resolver helpers are deterministic and side-effect free.
|
||||
- [ ] Context hook logic is idempotent on repeated runs.
|
||||
- [ ] Context hook exposes stable safe variables for reusable identifiers before templates consume them.
|
||||
- [ ] `[INDEX]` expansion creates concrete instance directories and removes placeholder template dir.
|
||||
|
||||
## G. End-to-end runs
|
||||
|
||||
- [ ] Fresh render into empty target directory succeeds.
|
||||
- [ ] Re-render into existing target with prior answers succeeds.
|
||||
- [ ] `--copier-force-copy true` path works.
|
||||
- [ ] Default/non-force path works.
|
||||
|
||||
## H. Documentation quality
|
||||
|
||||
- [ ] README explains local setup and one canonical bootstrap command.
|
||||
- [ ] Skill/instructions mention architecture and anti-patterns.
|
||||
- [ ] Examples use realistic paths and argument names.
|
||||
|
||||
## Exit criteria
|
||||
|
||||
You are done when all checklist sections pass and rendering is reproducible in both fresh and update scenarios.
|
||||
Reference in New Issue
Block a user