commit 97991021fff9e1f73202a82d29eea496ea0eb7aa Author: Oleg Lukasonok Date: Wed Aug 12 22:30:29 2026 +0300 template-engine-copier skill: initial import Copier + Jinja template-repository design skill: reference architecture, implementation workflow, validation checklist, and a full worked example template (questionnaire, extensions, resolvers, wrapper-first .scripts). Co-Authored-By: Claude Opus 5 (1M context) diff --git a/SKILL.md b/SKILL.md new file mode 100644 index 0000000..47a3cba --- /dev/null +++ b/SKILL.md @@ -0,0 +1,120 @@ +--- +name: template-engine-copier +description: Design and implement a Copier + Jinja based template repository, following the proven patterns used in iac-tmpl-aws-projects and iac-tmpl-aws-organizations. +--- + +# template-engine-copier + +Use this skill when you need to **create, refactor, or standardize** a repository that acts as a **Copier template engine** for downstream projects. + +This skill is based on real patterns from: + +- `iac-tmpl-aws-projects/` +- `iac-tmpl-aws-organizations/` + +and is meant to guide creation of repositories that have: + +1. clear questionnaire-driven inputs, +2. deterministic Jinja rendering, +3. conditional folder/file generation, +4. safe bootstrapping and re-render workflows. + +It now also includes a **concrete scaffold** you can copy from: + +- `example/` + +--- + +## Goal + +Deliver a template repository where operators can run one command like: + +```bash +task copier:bootstrap-one -- --copier-target-dir "../target-repo" --copier-params-file "./copier-answers.yaml" +``` + +and get a rendered repository with predictable structure and preserved answer history. + +--- + +## Read in this order + +1. `instructions/01-reference-architecture.md` +2. `instructions/02-implementation-workflow.md` +3. `instructions/03-validation-checklist.md` +4. `example/README.md` and `example/` + +--- + +## Canonical architecture (from analyzed repos) + +```text +/ + .copier/ + copier.yaml + questionnaire/*.yml + extensions/ + context_hook_extension/ # optional but recommended for complex templates + resolvers/ + .scripts/ + copier/ + Taskfile.yml + api/bootstrap-one.sh + lib/ + --env-vars-reader.sh + --index.sh + --index-api.sh + -bootstrap-one.sh + answers/ + --index-api.sh + -new-ensure-existance.sh + -previous-pull-and-merge.sh + -params-pull-and-merge.sh + base/ + loggers/ + Taskfile.yml + {{ _copier_conf.answers_file }}.jinja + {{ '.tmp-gitlab-ci.yml' }}.jinja + src/**/*.jinja +``` + +--- + +## Non-negotiable conventions + +1. Keep Copier source-of-truth in `.copier/copier.yaml`. +2. Keep questions separated under `.copier/questionnaire/*.yml` and include via `!include`. +3. Use `.jinja` suffix and render both file contents and file/folder names. +4. For dynamic instance folders, support `[INDEX]` expansion pattern via context hook. +5. Keep operational entrypoint wrapper-first: `Taskfile -> api -> lib`. +6. Merge answers in this order: + - previous target answers, + - explicit params file, + - final merged file used for `copier copy --data-file ...`. + +--- + +## Anti-patterns to avoid + +1. Putting all logic directly into one giant Jinja file. +2. Copying templates without answer merge strategy (breaks update flows). +3. Embedding one-off shell logic directly in Taskfile instead of API/lib wrappers. +4. Using ad-hoc variable names instead of stable normalized `flat.*` style for complex conditions. +5. Ignoring render-time safety for generated CI files (e.g., temp naming + controlled rename). + +--- + +## What to copy first (practical) + +When building a new template repo, copy these parts first from `example/`: + +1. `.copier/copier.yaml` +2. `.copier/questionnaire/0100-template.yml` +3. `.copier/extensions/context_hook_extension/hook.py` +4. `.copier/extensions/resolvers/resolve_safe_identifier.py` +5. `.scripts/copier/` (all files) +6. `Taskfile.yml` +7. `{{ _copier_conf.answers_file }}.jinja` +8. `{{ '.tmp-gitlab-ci.yml' }}.jinja` + +Then adapt `src/**/*.jinja` for your domain-specific resources. diff --git a/example/.copier/copier.yaml b/example/.copier/copier.yaml new file mode 100644 index 0000000..2cf49a7 --- /dev/null +++ b/example/.copier/copier.yaml @@ -0,0 +1,20 @@ +--- +_template_suffix: ".jinja" + +_exclude: + - ".git" + - ".copier" + - ".scripts" + - ".venv" + - "Taskfile.yml" + - "pyproject.toml" + - "copier-answers-new.yml" + +_jinja_extensions: + - copier_templates_extensions.TemplateExtensionLoader + - .copier/extensions/context_hook_extension/hook.py:ContextHookExtension + - .copier/extensions/resolvers/resolve_safe_identifier.py:SafeIdentifierExtension + +_answers_file: copier-answers.yml + +!include .copier/questionnaire/*.yml diff --git a/example/.copier/extensions/context_hook_extension/hook.py b/example/.copier/extensions/context_hook_extension/hook.py new file mode 100644 index 0000000..dd646fd --- /dev/null +++ b/example/.copier/extensions/context_hook_extension/hook.py @@ -0,0 +1,72 @@ +import os +import shutil + +from copier_templates_extensions import ContextHook + + +class ContextHookExtension(ContextHook): + update = False + + def hook(self, context): + _ensure_flat_variables(context) + _ensure_resources_instances(context) + return context + + +def _ensure_flat_variables(context): + if 'flat' not in context or not isinstance(context.get('flat'), dict): + context['flat'] = {} + + metadata = context.get('template_metadata', {}) if isinstance(context, dict) else {} + features = context.get('features', {}) if isinstance(context, dict) else {} + modules = features.get('modules', {}) if isinstance(features, dict) else {} + + domain_name = metadata.get('domain', '') if isinstance(metadata, dict) else '' + project_name = metadata.get('project', '') if isinstance(metadata, dict) else '' + + context['flat']['domain_name'] = domain_name + context['flat']['project_name'] = project_name + + instances = modules.get('instances', []) if isinstance(modules, dict) else [] + is_enabled = bool(modules.get('is_enabled', False)) if isinstance(modules, dict) else False + + names = [] + for instance in instances if isinstance(instances, list) else []: + if isinstance(instance, dict) and instance.get('name'): + names.append(f"{domain_name}-{project_name}-{instance['name']}") + + context['flat']['modules_is_enabled'] = is_enabled + context['flat']['module_instance_names'] = names + + +def _ensure_resources_instances(context): + """Expand folders containing [INDEX] into concrete per-instance directories.""" + base_path = './src' + if not os.path.exists(base_path): + return + + targets = [] + for root, dirs, _ in os.walk(base_path): + for d in dirs: + if '[INDEX]' in d: + targets.append(os.path.join(root, d)) + + for path in targets: + parent = os.path.dirname(path) + instances = context.get('flat', {}).get('module_instance_names', []) + for idx, name in enumerate(instances): + dst = os.path.join(parent, name) + if os.path.exists(dst): + continue + os.makedirs(dst, exist_ok=True) + for fn in os.listdir(path): + src_file = os.path.join(path, fn) + if not os.path.isfile(src_file): + continue + with open(src_file, 'r', encoding='utf-8') as f: + content = f.read().replace('INDEX', str(idx)) + with open(os.path.join(dst, fn), 'w', encoding='utf-8') as f: + f.write(content) + + if os.path.exists(path): + shutil.rmtree(path) diff --git a/example/.copier/extensions/resolvers/resolve_safe_identifier.py b/example/.copier/extensions/resolvers/resolve_safe_identifier.py new file mode 100644 index 0000000..979cba1 --- /dev/null +++ b/example/.copier/extensions/resolvers/resolve_safe_identifier.py @@ -0,0 +1,17 @@ +from jinja2.ext import Extension + + +def resolve_safe_identifier(name: str) -> str: + return ( + name + .replace("-", "_") + .replace("{", "") + .replace("}", "") + .replace(".", "_") + ) + + +class SafeIdentifierExtension(Extension): + def __init__(self, environment): + super().__init__(environment) + environment.globals["resolve_safe_identifier"] = resolve_safe_identifier diff --git a/example/.copier/packages/utils_loggers/__init__.py b/example/.copier/packages/utils_loggers/__init__.py new file mode 100644 index 0000000..00e2fa9 --- /dev/null +++ b/example/.copier/packages/utils_loggers/__init__.py @@ -0,0 +1 @@ +from .loggers import info, warning, debug diff --git a/example/.copier/packages/utils_loggers/loggers.py b/example/.copier/packages/utils_loggers/loggers.py new file mode 100644 index 0000000..ac59f85 --- /dev/null +++ b/example/.copier/packages/utils_loggers/loggers.py @@ -0,0 +1,16 @@ +import logging + + +logging.basicConfig(level=logging.INFO) + + +def info(name, message, extra=None): + logging.getLogger(name).info(message) + + +def warning(name, message, extra=None): + logging.getLogger(name).warning(message) + + +def debug(name, message, extra=None): + logging.getLogger(name).debug(message) diff --git a/example/.copier/questionnaire/0100-template.yml b/example/.copier/questionnaire/0100-template.yml new file mode 100644 index 0000000..aa3476a --- /dev/null +++ b/example/.copier/questionnaire/0100-template.yml @@ -0,0 +1,18 @@ +template_metadata: + type: yaml + help: "Generic template metadata used by naming and rendering hooks" + default: | + domain: demo + project: sample-product + +features: + type: yaml + help: "Feature/module matrix for template rendering" + default: | + modules: + is_enabled: true + instances: + - name: users + version: 1.0.0 + - name: billing + version: 1.0.0 diff --git a/example/.scripts/copier/Taskfile.yml b/example/.scripts/copier/Taskfile.yml new file mode 100644 index 0000000..9696aaf --- /dev/null +++ b/example/.scripts/copier/Taskfile.yml @@ -0,0 +1,9 @@ +version: "3" + +tasks: + bootstrap-one: + desc: -- --copier-force-copy "" --copier-target-dir "" --copier-params-file "" + cmds: + - | + ./.scripts/copier/api/bootstrap-one.sh {{ .CLI_ARGS }} + silent: true diff --git a/example/.scripts/copier/api/bootstrap-one.sh b/example/.scripts/copier/api/bootstrap-one.sh new file mode 100644 index 0000000..a8d47b3 --- /dev/null +++ b/example/.scripts/copier/api/bootstrap-one.sh @@ -0,0 +1,7 @@ +#!/bin/bash + +set -e + +. ./.scripts/copier/lib/--index.sh "$@" + +_copier_bootstrap_one diff --git a/example/.scripts/copier/lib/--env-vars-reader.sh b/example/.scripts/copier/lib/--env-vars-reader.sh new file mode 100644 index 0000000..0ea84f5 --- /dev/null +++ b/example/.scripts/copier/lib/--env-vars-reader.sh @@ -0,0 +1,41 @@ +#!/bin/bash + +export HL_V1_COPIER_TARGET_DIR="" +export HL_V1_COPIER_FORCE_COPY=false +export HL_V1_COPIER_PARAMS_FILE="" +export HL_V1_COPIER_FILE_COPIER_ANSWERS="copier-answers.yml" +export HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW="copier-answers-new.yml" + +ALL_ARGS=("$@") +while [ "$#" -gt 0 ]; do + case "$1" in + --copier-target-dir=*) + export HL_V1_COPIER_TARGET_DIR="${1#*=}" + shift + ;; + --copier-target-dir) + export HL_V1_COPIER_TARGET_DIR="$2" + shift 2 + ;; + --copier-force-copy=*) + export HL_V1_COPIER_FORCE_COPY="${1#*=}" + shift + ;; + --copier-force-copy) + export HL_V1_COPIER_FORCE_COPY="$2" + shift 2 + ;; + --copier-params-file=*) + export HL_V1_COPIER_PARAMS_FILE="${1#*=}" + shift + ;; + --copier-params-file) + export HL_V1_COPIER_PARAMS_FILE="$2" + shift 2 + ;; + *) + shift + ;; + esac +done +set -- "${ALL_ARGS[@]}" diff --git a/example/.scripts/copier/lib/--index-api.sh b/example/.scripts/copier/lib/--index-api.sh new file mode 100644 index 0000000..6e1da9d --- /dev/null +++ b/example/.scripts/copier/lib/--index-api.sh @@ -0,0 +1,4 @@ +#!/bin/bash + +. ./.scripts/copier/lib/answers/--index-api.sh +. ./.scripts/copier/lib/-bootstrap-one.sh diff --git a/example/.scripts/copier/lib/--index.sh b/example/.scripts/copier/lib/--index.sh new file mode 100644 index 0000000..ff70431 --- /dev/null +++ b/example/.scripts/copier/lib/--index.sh @@ -0,0 +1,4 @@ +#!/bin/bash + +. ./.scripts/copier/lib/--env-vars-reader.sh "$@" +. ./.scripts/copier/lib/--index-api.sh diff --git a/example/.scripts/copier/lib/-bootstrap-one.sh b/example/.scripts/copier/lib/-bootstrap-one.sh new file mode 100644 index 0000000..66af742 --- /dev/null +++ b/example/.scripts/copier/lib/-bootstrap-one.sh @@ -0,0 +1,33 @@ +#!/bin/bash + +set -e + +_copier_bootstrap_one() { + [ -z "${HL_V1_COPIER_TARGET_DIR}" ] && { + echo "--copier-target-dir is required" + return 1 + } + + _copier_answers_new_ensure_existence + _copier_answers_previous_pull_and_merge + _copier_answers_params_pull_and_merge + + if [ "${HL_V1_COPIER_FORCE_COPY}" = "true" ]; then + copier copy \ + --trust \ + --force \ + --overwrite \ + --data-file "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}" \ + . "${HL_V1_COPIER_TARGET_DIR}" + else + copier copy \ + --trust \ + --data-file "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}" \ + . "${HL_V1_COPIER_TARGET_DIR}" + fi + + # Keep template repo CI separate from rendered repo CI, same pattern as source repos. + if [ -f "${HL_V1_COPIER_TARGET_DIR}/.tmp-gitlab-ci.yml" ]; then + mv "${HL_V1_COPIER_TARGET_DIR}/.tmp-gitlab-ci.yml" "${HL_V1_COPIER_TARGET_DIR}/.gitlab-ci.yml" + fi +} diff --git a/example/.scripts/copier/lib/answers/--index-api.sh b/example/.scripts/copier/lib/answers/--index-api.sh new file mode 100644 index 0000000..898bb9f --- /dev/null +++ b/example/.scripts/copier/lib/answers/--index-api.sh @@ -0,0 +1,5 @@ +#!/bin/bash + +. ./.scripts/copier/lib/answers/-new-ensure-existance.sh +. ./.scripts/copier/lib/answers/-previous-pull-and-merge.sh +. ./.scripts/copier/lib/answers/-params-pull-and-merge.sh diff --git a/example/.scripts/copier/lib/answers/-new-ensure-existance.sh b/example/.scripts/copier/lib/answers/-new-ensure-existance.sh new file mode 100644 index 0000000..fa5483d --- /dev/null +++ b/example/.scripts/copier/lib/answers/-new-ensure-existance.sh @@ -0,0 +1,5 @@ +#!/bin/bash + +_copier_answers_new_ensure_existence() { + [ -f "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}" ] || touch "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}" +} diff --git a/example/.scripts/copier/lib/answers/-params-pull-and-merge.sh b/example/.scripts/copier/lib/answers/-params-pull-and-merge.sh new file mode 100644 index 0000000..c06a072 --- /dev/null +++ b/example/.scripts/copier/lib/answers/-params-pull-and-merge.sh @@ -0,0 +1,17 @@ +#!/bin/bash + +_copier_answers_params_pull_and_merge() { + [ -f "${HL_V1_COPIER_PARAMS_FILE}" ] || return 0 + [ -f "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}" ] || return 0 + + if command -v yq >/dev/null 2>&1; then + local TMP_WORK_FILE="tmp-work-file.yml" + yq eval-all '. as $item ireduce ({}; . * $item)' \ + "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}" \ + "${HL_V1_COPIER_PARAMS_FILE}" >"${TMP_WORK_FILE}" + mv "${TMP_WORK_FILE}" "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}" + else + # Fallback: params file wins as full override when yq is unavailable. + cp "${HL_V1_COPIER_PARAMS_FILE}" "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}" + fi +} diff --git a/example/.scripts/copier/lib/answers/-previous-pull-and-merge.sh b/example/.scripts/copier/lib/answers/-previous-pull-and-merge.sh new file mode 100644 index 0000000..d52d522 --- /dev/null +++ b/example/.scripts/copier/lib/answers/-previous-pull-and-merge.sh @@ -0,0 +1,6 @@ +#!/bin/bash + +_copier_answers_previous_pull_and_merge() { + local PREVIOUS="${HL_V1_COPIER_TARGET_DIR}/${HL_V1_COPIER_FILE_COPIER_ANSWERS}" + [ -f "${PREVIOUS}" ] && cp "${PREVIOUS}" "${HL_V1_COPIER_FILE_COPIER_ANSWERS_NEW}" || true +} diff --git a/example/README.md b/example/README.md new file mode 100644 index 0000000..17b340f --- /dev/null +++ b/example/README.md @@ -0,0 +1,16 @@ +# Example Template Repository + +This is a minimal but concrete Copier + Jinja template repository scaffold. + +It is intentionally **domain-generic** and demonstrates: + +- metadata + features questionnaire model, +- dynamic module/component folder generation, +- safe identifier resolver, +- shell-based bootstrap/answers merge flow. + +## Render example + +```bash +task copier:bootstrap-one -- --copier-target-dir "../rendered-example" --copier-params-file "./copier-answers.example.yaml" +``` diff --git a/example/Taskfile.yml b/example/Taskfile.yml new file mode 100644 index 0000000..4a42418 --- /dev/null +++ b/example/Taskfile.yml @@ -0,0 +1,4 @@ +version: "3" + +includes: + copier: ./.scripts/copier/Taskfile.yml diff --git a/example/copier-answers.example.yaml b/example/copier-answers.example.yaml new file mode 100644 index 0000000..6b6cddf --- /dev/null +++ b/example/copier-answers.example.yaml @@ -0,0 +1,12 @@ +template_metadata: + domain: demo + project: sample-product + +features: + modules: + is_enabled: true + instances: + - name: users + version: 1.0.0 + - name: billing + version: 1.0.0 diff --git a/example/pyproject.toml b/example/pyproject.toml new file mode 100644 index 0000000..ec5b1bd --- /dev/null +++ b/example/pyproject.toml @@ -0,0 +1,8 @@ +[project] +name = "example-template-repo" +version = "0.1.0" +requires-python = ">=3.10" +dependencies = [ + "copier==9.14.0", + "copier-templates-extensions==0.3.2", +] diff --git a/example/src/{{ 'modules' if flat.modules_is_enabled }}/index.txt.jinja b/example/src/{{ 'modules' if flat.modules_is_enabled }}/index.txt.jinja new file mode 100644 index 0000000..d46edd5 --- /dev/null +++ b/example/src/{{ 'modules' if flat.modules_is_enabled }}/index.txt.jinja @@ -0,0 +1,21 @@ +{% macro render() %} +{% set metadata = template_metadata if template_metadata is mapping else {} %} +{% set features_map = features if features is mapping else {} %} +{% set modules = features_map.get('modules', {}) if features_map else {} %} + +Template Domain: {{ metadata.get('domain', 'unknown-domain') }} +Template Project: {{ metadata.get('project', 'unknown-project') }} + +{% if modules.get('is_enabled', false) -%} +Include module components from ./components/ +{% else -%} +No modules enabled. +{% endif %} +{% endmacro %} + +{%- set result = render() -%} +{%- for line in result.split('\n') -%} +{% if line | trim | length != 0 -%} +{{ line }} +{% endif %} +{%- endfor %} diff --git a/example/src/{{ 'modules' if flat.modules_is_enabled }}/{{ 'components' if flat.modules_is_enabled }}/index.txt.jinja b/example/src/{{ 'modules' if flat.modules_is_enabled }}/{{ 'components' if flat.modules_is_enabled }}/index.txt.jinja new file mode 100644 index 0000000..ca2f74e --- /dev/null +++ b/example/src/{{ 'modules' if flat.modules_is_enabled }}/{{ 'components' if flat.modules_is_enabled }}/index.txt.jinja @@ -0,0 +1,5 @@ +{% set instances_names = flat.get('module_instance_names', []) %} + +{% for item in instances_names %} +component: {{ resolve_safe_identifier(item) }} +{% endfor %} diff --git a/example/src/{{ 'modules' if flat.modules_is_enabled }}/{{ 'components' if flat.modules_is_enabled }}/{{ flat.module_instance_names[INDEX] }}/main.txt.jinja b/example/src/{{ 'modules' if flat.modules_is_enabled }}/{{ 'components' if flat.modules_is_enabled }}/{{ flat.module_instance_names[INDEX] }}/main.txt.jinja new file mode 100644 index 0000000..5908386 --- /dev/null +++ b/example/src/{{ 'modules' if flat.modules_is_enabled }}/{{ 'components' if flat.modules_is_enabled }}/{{ flat.module_instance_names[INDEX] }}/main.txt.jinja @@ -0,0 +1,3 @@ +component_id={{ resolve_safe_identifier(flat.module_instance_names[INDEX]) }} +component_name={{ flat.module_instance_names[INDEX] }} +generated_with=copier+jinja diff --git a/example/{{ '.tmp-gitlab-ci.yml' }}.jinja b/example/{{ '.tmp-gitlab-ci.yml' }}.jinja new file mode 100644 index 0000000..dfb1d86 --- /dev/null +++ b/example/{{ '.tmp-gitlab-ci.yml' }}.jinja @@ -0,0 +1,17 @@ +stages: + - lint + - test + +template-content-check: + stage: lint + image: alpine:3.20 + script: + - test -f README.md + - test -f copier-answers.example.yaml + - test -d src + +template-smoke-test: + stage: test + image: alpine:3.20 + script: + - echo "Template repository scaffold is present" diff --git a/example/{{ _copier_conf.answers_file }}.jinja b/example/{{ _copier_conf.answers_file }}.jinja new file mode 100644 index 0000000..69141bb --- /dev/null +++ b/example/{{ _copier_conf.answers_file }}.jinja @@ -0,0 +1,2 @@ +# Changes here will be overwritten by Copier; NEVER EDIT MANUALLY +{{ _copier_answers|to_nice_yaml -}} diff --git a/instructions/01-reference-architecture.md b/instructions/01-reference-architecture.md new file mode 100644 index 0000000..478069c --- /dev/null +++ b/instructions/01-reference-architecture.md @@ -0,0 +1,218 @@ +# Copier + Jinja Template Engine Reference Architecture + +This reference architecture is synthesized from: + +- `iac-tmpl-aws-projects/` +- `iac-tmpl-aws-organizations/` + +and should be treated as the default blueprint for new template repositories. + +## 1) Copier core contract + +Use `.copier/copier.yaml` as central control plane. + +### Recommended shape + +```yaml +_template_suffix: ".jinja" + +_exclude: + - ".git" + - ".copier" + - ".scripts" + - ".venv" + - ".devbox" + - "devbox.json" + - "devbox.lock" + - ".gitlab-ci.yml" + - "Taskfile.yml" + - "pyproject.toml" + - "uv.lock" + - "copier-answers-new.yml" + - "copier-answers.example.yaml" + +_jinja_extensions: + - copier_templates_extensions.TemplateExtensionLoader + # plus custom extension(s) + +_answers_file: copier-answers.yml + +!include .copier/questionnaire/*.yml +``` + +## 2) Questionnaire strategy + +Keep questions modular under `.copier/questionnaire/`. + +Why: + +- supports domain separation, +- scales better than one monolithic YAML, +- makes review and versioning easier. + +## 3) Context shaping strategy + +For advanced templates (especially conditional directories + repeated resource modules), use custom context hook extension. + +Pattern used in `iac-tmpl-aws-projects`: + +- `.copier/extensions/context_hook_extension/hook.py` +- merged orchestration in `hook_function.py` + 1. compute/normalize flat variables, + 2. expand `[INDEX]` instance folders. + +### Why `flat.*` variables + +They reduce deep nested path usage in Jinja and make conditional file/folder naming concise: + +- `flat.dev_is_enabled` +- `flat.dev_vm_is_enabled` +- `flat.dev_vm_instances_names` + +They should also carry any normalized/safe identifiers needed by templates: + +- `flat.project_safe` +- `flat.environment_safe` +- `flat.component_safe_names` + +Rule: do **not** repeat normalization logic such as `lower | replace(...) | trim` throughout template paths or file bodies. If a safe identifier is needed more than once, calculate it once in the context hook (or a deterministic resolver) and expose it as a stable context variable like `flat.project_safe`. + +## 4) Dynamic folder generation (`[INDEX]`) + +In `iac-tmpl-aws-projects`, folders like: + +```text +iac/{{ 'dev' if flat.dev_is_enabled }}/.../{{ flat.dev_vm_instances_names[INDEX] }}/ +``` + +are expanded by context hook functions that: + +1. find `[INDEX]` patterns, +2. map pattern -> context list, +3. create concrete per-instance folders, +4. replace `INDEX` placeholders in contents, +5. remove the template placeholder folder. + +## 5) Custom Jinja resolvers + +Keep custom Jinja helpers in `.copier/extensions/resolvers/` and register them in `_jinja_extensions`. + +Examples discovered: + +- `resolve_safe_tf_module_name(...)` +- `resolve_regions(...)` +- `resolve_availability_zones(...)` +- tags resolver(s) for project/organization context. + +Rule: put transformation logic in extensions, not inline repeated Jinja blocks. + +For safe names specifically, prefer context-hook derived variables over inline filter chains. Templates should read: + +```text +applications/{{ flat.project_safe }}/... +``` + +not: + +```text +applications/{{ project.name | lower | replace(' ', '-') | replace('_', '-') }}/... +``` + +## 6) Operational wrapper architecture + +Use shell wrapper-first command surface: + +`Taskfile -> api -> lib` + +Concrete layout: + +- `.scripts/copier/Taskfile.yml` - operator tasks +- `.scripts/copier/api/bootstrap-one.sh` - thin wrapper +- `.scripts/copier/lib/*.sh` - actual behavior + +This keeps CLI UX stable while allowing internal evolution. + +### Bootstrap from a temporary render workspace + +Follow the `iac-tmpl-aws-projects` runtime pattern: the public API wrapper should not render directly from the live template repo folder. It should first create a sibling temporary copy and execute rendering from there. + +Canonical flow: + +1. Capture the live template folder name from `$PWD`. +2. Remove any previous sibling temp folder, e.g. `../tmp-${TEMPLATE_FOLDER}`. +3. Copy the live template folder to `../tmp-${TEMPLATE_FOLDER}`. +4. `cd` into the temp folder. +5. Load logger and copier indexes from the temp folder. +6. Run `_copier_bootstrap_one` from the temp folder. +7. Return to the live template folder after rendering. + +Inside `_copier_bootstrap_one`, before invoking `copier copy`: + +1. Copy `.copier/copier.yaml` to root `copier.yaml` when the template stores Copier config under `.copier/`. +2. Remove temp-copy internals that interfere with Copier local rendering: + - `.git` + - `.devbox` + - `.venv` +3. Then run `copier copy --trust --data-file ... . "$TARGET_DIR"` from the temp folder. + +Why this matters: + +- Copier sees a plain local template directory, not the live Git repository. +- It avoids Git/tag discovery noise such as `No git tags found in template; using HEAD as ref`. +- It prevents the render from mutating or deleting files in the live template repo. +- It keeps generated transient files (`copier.yaml`, `copier-answers-new.yml`, temp merge files) isolated in the temp copy. + +Do not solve local bootstrap Git warnings by creating tags just to satisfy Copier. Tags are for intentional template releases, not for local render hygiene. + +## 7) Answers merge pipeline + +Use 3-step pre-render merge flow before `copier copy`: + +1. ensure `copier-answers-new.yml` exists, +2. pull previous target answers (`/copier-answers.yml`) if present, +3. merge explicit params file into the working answers file. + +Then execute: + +```bash +copier copy --trust --data-file "copier-answers-new.yml" ... +``` + +## 8) Generated CI file safety pattern + +When template repo itself has CI files that must not be copied literally, use a temporary rendered filename: + +- template file: `{{ '.tmp-gitlab-ci.yml' }}.jinja` +- post-render rename in bootstrap lib: + - `.tmp-gitlab-ci.yml` -> `.gitlab-ci.yml` + +## 9) Dependency baseline + +Keep Copier stack pinned in `pyproject.toml`: + +- `copier` +- `copier-templates-extensions` +- supportive tooling (e.g. `check-jsonschema`, logging libs) + +Use lockfile (`uv.lock`) for deterministic local behavior. + +## 10) Where this is implemented in this skill + +See concrete scaffold: + +- `example/.copier/copier.yaml` +- `example/.copier/questionnaire/0100-template.yml` +- `example/.copier/extensions/context_hook_extension/hook.py` +- `example/.copier/extensions/resolvers/resolve_safe_identifier.py` +- `example/.scripts/copier/` + + +## 11) Environment variable prefix convention + +Before implementing shell wrappers, **propose and confirm** the environment variable prefix with the user (e.g. `HL_V1_`, `PALANTIR_`, etc.). + +Rules: + +- Do not hardcode an inherited prefix from old scaffolds without confirmation. +- Keep one prefix consistently across `--env-vars-reader.sh`, bootstrap libs, and answer merge libs. +- Document the chosen prefix in README or implementation notes. diff --git a/instructions/02-implementation-workflow.md b/instructions/02-implementation-workflow.md new file mode 100644 index 0000000..7fba891 --- /dev/null +++ b/instructions/02-implementation-workflow.md @@ -0,0 +1,156 @@ +# Implementation Workflow (Copier + Jinja Template Repository) + +Follow this sequence when creating a new template-engine repository. + +## Phase 1 — Define rendering contract + +1. Define template scope (what downstream repository you generate). +2. Identify stable input schema sections (e.g. tenant/project/environments/resources). +3. Decide what is: + - static files, + - conditional files/folders, + - repeatable per-instance blocks. + +Deliverable: first draft of `copier-answers.example.yaml`. + +## Phase 2 — Bootstrap Copier skeleton + +Create: + +- `.copier/copier.yaml` +- `.copier/questionnaire/*.yml` +- `{{ _copier_conf.answers_file }}.jinja` + +Use baseline: + +- `_template_suffix: ".jinja"` +- modular `!include .copier/questionnaire/*.yml` +- explicit `_exclude` list +- `_answers_file: copier-answers.yml` + +## Phase 3 — Create operational wrappers + +Implement wrapper chain: + +1. root `Taskfile.yml` includes `.scripts/copier/Taskfile.yml` +2. `.scripts/copier/Taskfile.yml` exposes `bootstrap-one` +3. `.scripts/copier/api/bootstrap-one.sh` remains thin +4. **confirm ENV var prefix with user** (or propose one) before implementing lib internals +5. `.scripts/copier/lib/` contains logic and answers merge helpers + +Minimum behavior in lib: + +- parse CLI args with both styles: `--arg value` and `--arg=value` (for `--copier-target-dir`, `--copier-force-copy`, `--copier-params-file`), +- prepare merged answers file, +- run `copier copy --trust --data-file ...` from a sibling temporary copy of the template repo, not from the live template repo folder, +- post-render sanitization/renames if needed. + +Mandatory temp-copy bootstrap pattern from `iac-tmpl-aws-projects`: + +- In `api/bootstrap-one.sh`: + 1. derive the live template folder name from `basename "$PWD"`, + 2. delete any old sibling `../tmp-${TEMPLATE_FOLDER}`, + 3. copy the whole template repo into `../tmp-${TEMPLATE_FOLDER}`, + 4. `cd` into the temp folder, + 5. source `.scripts/loggers/lib/--index.sh` and `.scripts/copier/lib/--index.sh`, + 6. execute `_copier_bootstrap_one`, + 7. return to the original live template folder. +- In `lib/-bootstrap-one.sh`, before `copier copy`: + 1. copy `.copier/copier.yaml` to root `copier.yaml` if the repo uses `.copier/copier.yaml`, + 2. remove `.git`, `.devbox`, and `.venv` from the temp copy, + 3. run Copier from the temp folder into the real target. + +Do **not** create Git tags merely to silence Copier's local template version warning. The preferred local bootstrap hygiene is rendering from the sanitized temp copy. Use `--vcs-ref=HEAD` only when there is an explicit reason to render directly from a Git-backed live folder. + +## Phase 4 — Add Jinja template layers + +1. Add top-level templates (`iac/main.tf.jinja`, providers, backend, etc.). +2. Gate directories via folder-name expressions: + - `{{ 'dev' if flat.dev_is_enabled }}` +3. Gate module calls inside files via `{% if ... %}`. + +Pattern: + +- folder condition controls existence, +- inner condition controls file content clarity. + +**Mandatory wrapped render-path rule:** + +- For every optional module/resource, apply conditions at the **path level** (folder/file names), not only inside file contents. +- Preferred pattern (from `iac-tmpl-aws-projects`): + - `iac/{{ 'dev' if flat.dev_is_enabled }}/{{ 'compute' if flat.dev_compute_is_enabled }}/main.tf.jinja` +- Apply this systematically to all optional Jinja outputs so disabled features do not produce placeholder files/directories. + +**Mandatory safe-variable rule:** + +- Do not embed repeated normalization/sanitization logic in template paths or file bodies, e.g. avoid: + - `{{ project.name | lower | replace(' ', '-') | replace('_', '-') | replace('.', '-') }}` +- If a safe name is needed, calculate it once in the context hook and expose a named variable, e.g.: + - `flat.project_safe` + - `flat.environment_safe` + - `flat.component_safe_names` +- Use the named variable in templates: + - `applications/{{ flat.project_safe }}/main.yaml.jinja` +- If a one-off transformation is genuinely needed, prefer a small deterministic resolver over duplicating filter chains. + +## Phase 5 — Add custom extensions (if complexity requires) + +When logic gets repetitive, move it into extensions. + +### Resolver extension examples + +- normalize names: `resolve_safe_tf_module_name(name)` +- derive tags / regions / zones + +### Context hook extension examples + +- calculate `flat.*` shortcuts, +- calculate safe/normalized names such as `flat.project_safe` and `flat.environment_safe`, +- expand `[INDEX]` directories into per-instance folders. + +Register in `.copier/copier.yaml` under `_jinja_extensions`. + +## Phase 6 — Implement answers merge strategy + +Use this exact order to avoid destructive updates: + +1. `copier-answers-new.yml` create if missing, +2. copy `/copier-answers.yml` if present, +3. merge params file into working answers file (`yq eval-all ...`). + +Then run copier using merged file. + +## Phase 7 — Handle template/self-repo conflicts + +If template repo contains files that should not be copied 1:1 (e.g. self CI): + +1. render as temp name (`.tmp-gitlab-ci.yml`), +2. rename in target during bootstrap. + +## Phase 8 — Add docs and examples + +Add/update: + +- `README.md` with local setup and bootstrap command, +- `copier-answers.example.yaml` with realistic schema examples, +- optional `bootstrap-one-test` task for local maintainer testing. + +## Phase 9 — Validate end-to-end + +Run two critical scenarios: + +1. fresh render into empty target, +2. re-render into existing target with existing `copier-answers.yml`. + +Both must succeed without manual edits. + +## Quick-start from this skill example + +If you want to accelerate implementation, copy `example/` and then: + +1. expand questionnaire files, +2. expand `flat.*` derivation logic in context hook, +3. add domain-specific resolver extensions, +4. extend `iac/**/*.jinja` resource tree, +5. keep bootstrap/answers flow unchanged unless you have a strong reason. + diff --git a/instructions/03-validation-checklist.md b/instructions/03-validation-checklist.md new file mode 100644 index 0000000..f637dfe --- /dev/null +++ b/instructions/03-validation-checklist.md @@ -0,0 +1,75 @@ +# Validation Checklist (Template Engine Repository) + +Use this checklist before considering Copier/Jinja template work complete. + +## A. Copier configuration + +- [ ] `.copier/copier.yaml` exists and is valid YAML. +- [ ] `_template_suffix` is set to `.jinja`. +- [ ] `_answers_file` is explicitly configured. +- [ ] `_exclude` blocks repo-internal operational files (`.copier`, `.scripts`, env/lock files as needed). +- [ ] Questionnaire is loaded via `!include .copier/questionnaire/*.yml`. + +## B. Input schema and answers + +- [ ] `copier-answers.example.yaml` reflects real expected shape. +- [ ] Required keys used in templates are documented in the example answers. +- [ ] `{{ _copier_conf.answers_file }}.jinja` renders `_copier_answers|to_nice_yaml`. + +## C. Wrapper runtime surface + +- [ ] Root `Taskfile.yml` includes copier module Taskfile. +- [ ] `.scripts/copier/Taskfile.yml` has an operator-friendly `bootstrap-one` task. +- [ ] API wrapper (`api/bootstrap-one.sh`) creates and enters a sibling `tmp-` copy before sourcing libs and rendering. +- [ ] API wrapper returns to the live template repo after the temp-copy render completes. +- [ ] ENV variable prefix is explicitly confirmed/proposed and consistently applied across wrapper libs. +- [ ] Lib layer parses CLI args for: +- [ ] Lib layer parses CLI args for **both** forms: `--arg value` and `--arg=value`. + - [ ] `--copier-target-dir` + - [ ] `--copier-force-copy` + - [ ] `--copier-params-file` + +## D. Answers merge behavior + +- [ ] `copier-answers-new.yml` is created if missing. +- [ ] Existing target answers are imported when present. +- [ ] Params file is merged into working answers file. +- [ ] Render uses `--data-file` pointing to merged answers file. +- [ ] Local bootstrap render runs from the sanitized temp copy, not directly from the live Git-backed template folder. +- [ ] Temp copy removes `.git`, `.devbox`, and `.venv` before `copier copy` to avoid Copier/Git interference. +- [ ] If Copier config lives under `.copier/copier.yaml`, temp-copy bootstrap copies it to root `copier.yaml` before rendering. +- [ ] No Git tags are created merely to suppress Copier local render warnings. + +## E. Templating behavior + +- [ ] Conditional folder names render correctly (e.g. `{{ 'dev' if ... }}`). +- [ ] **All optional resources** are path-wrapped with Jinja conditions (folder and/or file names), not only guarded in file bodies. +- [ ] Conditional Jinja blocks in files match folder-level conditions. +- [ ] Safe/normalized names are provided by context-hook or resolver variables (e.g. `flat.project_safe`), not repeated inline filter chains. +- [ ] Template paths and file bodies do not duplicate complex expressions like `lower | replace(...) | trim` for identifiers used in more than one place. +- [ ] Any generated temporary filenames (e.g. `.tmp-gitlab-ci.yml`) are renamed correctly post-render. + +## F. Extensions (if used) + +- [ ] All custom extensions are registered in `_jinja_extensions`. +- [ ] Resolver helpers are deterministic and side-effect free. +- [ ] Context hook logic is idempotent on repeated runs. +- [ ] Context hook exposes stable safe variables for reusable identifiers before templates consume them. +- [ ] `[INDEX]` expansion creates concrete instance directories and removes placeholder template dir. + +## G. End-to-end runs + +- [ ] Fresh render into empty target directory succeeds. +- [ ] Re-render into existing target with prior answers succeeds. +- [ ] `--copier-force-copy true` path works. +- [ ] Default/non-force path works. + +## H. Documentation quality + +- [ ] README explains local setup and one canonical bootstrap command. +- [ ] Skill/instructions mention architecture and anti-patterns. +- [ ] Examples use realistic paths and argument names. + +## Exit criteria + +You are done when all checklist sections pass and rendering is reproducible in both fresh and update scenarios.