k9-ownership-byod-setup

Merge pull request #113 from tillwf/revert-judge-hill-climb-delegation Revert the hill-climb delegation and its holdout publish gate

Revert "Delegate the judge hill-climb to auto-experiment" This reverts commit 6a89a7dddea112dd03d8d6db575f938f1ca11adb.

Revert "Gate publishing on the holdout, not the fitted score" This reverts commit f0febd98342a2feae7bdd6d71ce9c9c7af431630.

Merge pull request #109 from tillwf/till.wohlfarth/auto-labelling-skill Add agent-observability-build-eval-from-annotations skill

Gate publishing on the holdout, not the fitted score Ran the delegated loop end to end on a real 89-row queue. Iteration 1 improved the fitted F1 from 0.255 to 0.485 at |t| = 11.70 — the most confident result of the run, comfortably past both the significance threshold and min_delta — and scored 0.000 on the holdout, catching none of its three positives. A control judge from an earlier run scored 0.667 on the same rows through the same harness, so the harness was sound and the judge had simply overfitted 7 training positives. The publish gate said "never beat the constant-class baseline" without naming a split, which would have let that judge ship. It now names the holdout, and adds the case the run also hit: the new judge was worse than the evaluator already deployed, and a write is a full replace with no version history, so shipping it would have silently destroyed a better one. Co-Authored-By: Claude Opus 5

Delegate the judge hill-climb to auto-experiment build-eval-from-annotations and auto-experiment ran the same loop — baseline, census the errors, one focused change, re-score, keep what improved — on different objects, and build-eval got there by restating the loop rather than reusing it. Its copy was the weaker one: `runs` and `min_delta` were never derived from measured noise, there was no cost estimate before spending it, no per-iteration reporting, and an informal census. PR #108 closed most of the remaining distance by teaching auto-experiment to read an annotation queue, which duplicated this skill's Phase 1 almost line for line. So build-eval now keeps only what is actually its own — what the judge is for, what it may see, and publishing the winner — and hands the loop over. SKILL.md loses ~200 lines: the backend rules, the keep/discard policy, the mechanism audit, the stop conditions and the census all move to their single source, and the rubric points at auto-experiment's `_noise_policy` and `_mechanism_audit` rather than keeping a copy that would drift from the one the loop obeys. Two changes make auto-experiment drivable this way: - A caller may supply the harness (`harness_provided_by`). Step 2 validates it against the stdout contract instead of generating one, and STOPs if it does not conform — silently substituting a template harness would measure something other than what the caller asked for. - `mean` is the run's score under the configured metric; a row average is the default, not the definition. F1 on a class, macro-F1 and kappa are computed over the whole run's confusion matrix and cannot be recovered from a row average. That second one is not a technicality. Fitting a grader on a real 55/7 corpus produced a candidate scoring 0.855 accuracy that gained 14 rows, lost 3, and was the only statistically significant change in eleven iterations — while catching zero of the seven known errors. Every number about it was true and the judge was a rubber stamp. A loop gated on a row-averaged accuracy hill-climbs toward exactly that, so the rubric now requires the constant-class floor beside any headline the user picks. `references/judge_harness_template.py` is the harness: it re-reads the prompt under optimization every pass, runs the judge over the corpus `runs` times, and emits the agreed metric. Verified against the 89-row queue — mean 0.564, stdev 0.026 over 3 runs, reproducing the 0.545 holdout F1 the old majority-vote path measured, and finally giving the run-to-run spread that `min_delta` needs. What this gives up, said plainly in both files: independent full re-runs have no per-row majority and McNemar needs paired rows, so both are gone, replaced by mean +/- stdev and the two-sample t-test. McNemar was the stronger test for this data. That is a real trade of statistical power for a loop maintained in one place. build-eval no longer runs standalone; SKILL.md and README say so, and it STOPs rather than improvising a loop if auto-experiment is absent. Co-Authored-By: Claude Opus 5

Rename to build-eval-from-annotations, and fit it to a real queue The skill was verified against one queue shape: a single boolean label over `type: trace` content, with reviewer reasoning text, in an app with live production spans. Running it on a second queue — 89 rows, three required categorical labels, `type: experiment_trace`, no reasoning text, an app that only runs as experiments — broke four of those five assumptions, and everything here comes from making that run work. `auto-labelling` described the mechanism rather than the deliverable, so the skill is now `agent-observability-build-eval-from-annotations`, which says what goes in and what comes out. The state directory follows (`.build_eval_from_annotations/`). Old trigger phrases are kept as aliases. Verdict contract: `reasoning` and `confidence` are now default fields, locally and in the published evaluator. Confidence is a percentage — an integer 0-100, never a 0-1 probability — reported and calibrated against correctness, never used to decide. `87.5` rounds; `0.9` is flagged as ambiguous rather than silently promoted to 90; a missing or malformed confidence keeps its label rather than voiding the pass. Queue shapes beyond a boolean trace label: - `experiment_trace` content needs a resolver, not a span walk. The documented endpoint is recorded, including which of its two failures means "expired content" — the case that ends a run before it spends the corpus discovering it row by row. - Categorical values are lists, and rows can be multi-select. Comparison is now order-insensitive set equality, so `["a","b"]` and `["b","a"]` are one answer instead of a disagreement that also splits the majority vote. Partial credit is available (jaccard, similarity groups) but only from a taxonomy the user supplies. - Several labels per queue: ask whether to fit one joint judge or one per label, and score a joint verdict with `--label-field`. - `f1_minority` refuses a corpus with more than two classes instead of quietly making the rarest class positive. Ground truth got stricter. A dataset's `expected_output` is not it: on a real batch, a row whose output and expected_output were identical was failed by the reviewer on two of three labels. Only the human value is ground truth, and a prior model's prediction of the same label is leakage in a replicator run. Runner fixes, all found by running it: - `PAYLOAD_TEMPLATE.format()` crashed with `KeyError: '"label"'` — a JSON example added literal braces to a format string. Every judge run would have died on the first row. - With no `anthropic` SDK installed, the runner spawned a Node process per pass via `claude -p`; the OS killed two iterations outright. It now prefers a stdlib HTTPS call when a key is present, which also cut an iteration from 11 minutes to 90 seconds. - `temperature` is rejected outright by the newer models, so ask for determinism and fall back, recording that the run is then unpinned. Loop defaults: 10 iterations, and stop after 3 consecutive iterations worse than the best — measured on the point estimate, since significance is unreachable at these corpus sizes and waiting for it means never stopping. A kept-but-within-noise gain resets the counter; an unmeasured iteration neither resets nor advances it. Finally, the run ends by printing the evaluator's name and the Evaluations URL. `enabled: false` is a disabled evaluator, not a Datadog draft, and the skill no longer calls it one — a write is a full replace, so republishing a later iteration overwrites the earlier config with no history in the org. Co-Authored-By: Claude Opus 5

Note that the evaluator-write restriction is a rollout, not a capability The identical call with the identical payload was accepted on a us5 prod org and refused on datad0g.com, so the "versioned, edit in the UI" rejection is where a site sits in the rollout rather than something missing from the API. Staging runs ahead, so the refusal should be expected to spread rather than disappear, and the UI fallback is early rather than dead code. Probing settled it: a disabled throwaway evaluator pointed at a nonexistent ml_app was created on prod, read back, deleted, and confirmed gone. Co-Authored-By: Claude Opus 5

Handle sites that refuse API creation of evaluators Running Phase 8 against datad0g.com turned up a wall the skill had no answer for: `create_or_update_llmobs_evaluator` returns 400 "custom evaluator is versioned and can only be edited from the LLM Observability UI" - for a name that does not exist yet, and for any other name, so API creation is simply unavailable there. With the previous text, the run would have ended with the deliverable it now promises being impossible. So the skill keeps its promise a different way when that happens: verify nothing partial landed, write the full config to `.auto_labelling/evaluator_config.json` ready to paste into Evaluations -> New Evaluator, and report "no evaluator was created, because the API refuses it on this site; here is the UI-ready config". A delivered fallback with the blocker named, not a step that quietly vanished. Two more things the run established: - `output_schema` must be a bare JSON Schema on write. The `{name, schema, strict}` wrapper that `get_llmobs_evaluator` returns is rejected as `invalid BYOP output schema`, so a read cannot be round-tripped into a write without unwrapping it. - The judge provider has to come from what the org can actually call. The `integration_provider` enum does not cover every provider real evaluators use, and an org can hold a configured-but-broken integration - one was answering 401 from inside another evaluator's error field. Read an existing evaluator, and hand the choice to the user rather than guessing a model the org cannot reach. Co-Authored-By: Claude Opus 5

Make the evaluator the deliverable, not an optional last step Phase 8 was "publish, only on an explicit yes", which let a run end with a report and a prompt file on disk. Nobody can run that: a fitted judge is only useful once it exists as an evaluator the user can open, review and enable. So every run now ends by creating it, and the questions move: the name and target are confirmed with the user, the creation is not. It is still always written `enabled: false` - switching it on stays the user's call, in the UI, at whatever sampling they choose. Verification changes with it. A write that succeeded but does not appear in `list_llmobs_evals_by_ml_app` has not been delivered, because that listing is what backs the Evaluations page, so the check is the listing plus a read-back, and the run reports the URL and the eval_name to look for. Also records the trace-scoped span-selector template syntax (`{{spans[meta.span.kind:llm]...}}`) found while running the skill, and pins the report's headline to the deployable score rather than the fitted one. Finishing without an evaluator is still legitimate - too few labels, a judge that never beat the constant-class baseline, evidence no eval_scope can reach - but it is now reported as a failed run with the blocker named, not as a step that quietly did not happen. Co-Authored-By: Claude Opus 5

Add agent-observability-auto-labelling skill Fits an LLM-judge evaluator to the human labels already sitting in an annotation queue, by measurement rather than by inspection. The loop mirrors agent-observability-auto-experiment, with the judge as the object under optimization instead of the app, and a human label as the ground truth instead of a rubric: read the queue, work out where in the trace the labelled property actually lives, draft a judge, score it against the labelled rows with a metric agreed with the user, read its errors, change one thing, re-score, keep only what improved. Publishes the winner as a disabled evaluator draft, never enabled, and never writes predictions back into the queue. Shaped by a run against a real staging queue (13 labelled rows, boolean label, 12-span traces): - Phase 2 exists because the labelled property was not decidable inside the labelled trace at all - the binding user feedback was given in earlier turns of the session. The evidence map is the run's contract for what a datapoint is, and it has to be validated against the labels before any judging. - The gap between the fitted judge and a publishable evaluator is first-class: managed prompt templates resolve span selectors within one trace, so a judge fitted on cross-turn evidence must be re-scored on what the deployed evaluator would actually see. In that run: F1 1.000 fitted, 0.800 deployable. - Keep decisions use the point estimate plus a mechanism audit; McNemar only labels confidence. At 13 rows nothing is significant, so gating on significance would mean never moving, and the minority-class recall check is what stops a judge from drifting to the majority class. references/judge_runner.py runs the judge N times per row with a strict JSON contract; references/scoring.py computes the metric, confusion matrix, Wilson CI, flip rate and McNemar, stdlib only. rubrics.md holds the rules that must not bend: the label is ground truth, leakage is banned, unparseable is counted rather than coerced, and publishing is gated. Co-Authored-By: Claude Opus 5

Merge pull request #108 from tillwf/till.wohlfarth/auto-experiment-datadog-datasets auto-experiment: store the eval corpus in Datadog Datasets, and load it from an annotation queue

Add dd-orchestrator skill (#111) * Add dd-orchestrator skill * Fix dd-orchestrator review findings: dispatch gate, telemetry breaker, fetch isolation, trace framing Address 8 correctness/reliability issues raised in PR review: - resolve.py: an unresolved CHOICE_POINT now sets STOP_REASON=awaiting_choice even when a partial plan exists, so no product is silently dropped past the dispatch gate (e.g. "rum,apm" with no platform no longer reports STOP_REASON=none). - emit.py: the telemetry breaker now bounds a persistently down intake. Critical (resolve-core) events bypass an open breaker only until the 3-failure threshold, and terminal HTTP failures (4xx/5xx/429) trip the same breaker as transport failures -- previously a 503 retried 3x for every event indefinitely and critical events never stopped, adding a timeout per plan node. - fetch_skill.py: each fetch uses a fresh run-scoped temp dir, so a partial or older fetch cannot leave stale files to execute and concurrent fetches cannot interleave. - resolve.py: --trace no longer prints a duplicate unframed "SESSION ID:" line, so the trace is a single machine-readable block; SUGGESTED enrichments (e.g. a cloud connector) are now shown inside that block. - SKILL.md: remove the unbacked "[Subset: ...]" approve option (no resolver pass re-plans a dependency-safe subset); document dd-account-setup as the sole confirm-gate preflight exception; reword cloud connect as an optional suggestion (it is never a hard requirement anywhere in the catalog).

auto-experiment: load the eval corpus from an annotation queue The skill could only build a corpus from raw traces, a dataset, or a local file, so the one place in LLM-Obs that already holds human verdicts — an annotation queue — was unreachable. That is the strongest ground truth the platform has, and the rubric already prefers a deterministic metric over an LLM judge whenever ground truth exists. Adds `annotation_queue_id` as a data source, between `dataset_id` and the trace sources in the priority order: curated beats raw, an explicit dataset beats a queue. The queue's annotated interactions become the corpus, each `content_id` expanded through the existing trace path to get input/output, so the messages-source and data-selection rules apply unchanged. The resulting datapoints are written to a corpus dataset like any other trace-derived source, and the interaction UUID is the eval-set id (`content_id` is not — one trace can be queued twice). Which label means what cannot be inferred from an arbitrary user-defined schema, so `annotation_label_map` is a new must-ask intake field naming the label that supplies `expected_output` plus an optional filter label. Picking a label because its name looks right would invent the ground truth every score is measured against — the same failure the `evaluators` rule already forbids. Three exclusions are documented alongside the existing "no scoreable target span" rule, all for the same reason (no ground truth to score against): pending interactions, an empty mapped label, and interactions whose reviewers disagree. A disagreement is reported, never resolved by taking the newest. Backend mechanics were verified against pup 1.8.0 and the us1 MCP and differ enough to warrant their own section: only mcp filters server-side (`only_annotated`) and returns `annotated_count`/`pending_count`, while pup's `annotation-queues interactions list` takes no filter flags and returns no counts, so a pup run filters and counts client-side rather than hopping to MCP for one call. Also records that labels must be addressed by `label_schema_id` (`name_when_saved` drifts), that an unset label comes back `""`/`[]`, and that a queue's own `project_id` can be empty and so is not a safe derivation for the run's project. Co-Authored-By: Claude Opus 5 (1M context)

auto-experiment: store the eval corpus in Datadog Datasets The corpus and its val/test splits were local files (`data.jsonl`, `data.val.jsonl`, `data.test.jsonl`) committed into the user's repo. That made a trace-derived corpus unreproducible outside the scratch branch, put dataset content in git, and left nothing stopping a later iteration from re-splitting the rows every earlier score was measured on. Both splits are now Datadog LLM-Obs Datasets, created once per run with a single UTC timestamp in their names and referenced by id from config.json for every later iteration. `ml_app`/`trace_ids` sources get a corpus dataset of their own, so trace-derived runs are addressable afterwards; a `dataset_id` source is read, never written to. A `local_dataset_path` stays an exception: intake asks whether to upload it or keep it offline, and `local_file` mode keeps the old jsonl path (now gitignored). The harness has no MCP tools and cannot pay a re-download per eval pass, so the orchestrator hydrates the split it wants scored into an uncommitted `.auto_experiment/cache/.jsonl` and points the harness at it with the new `AUTO_EXP_DATASET_ID`. A missing cache is a hard error, never an empty eval set. `eval_results.jsonl` stays a committed local jsonl — it is this run's measurements, not corpus data, and the census and mechanism audit diff it against the best commit. Adds `project_id` as an intake field because every dataset write needs one on both backends, and documents the two write paths: the add_llmobs_dataset_records preview/confirm two-step on mcp, and pup's `datasets create` / `datasets batch-update`, whose --file bodies must be confirmed empirically rather than guessed from the table. Co-Authored-By: Claude Opus 5 (1M context)

fix(dd-account-setup): OAuth callback port, .env password clarity, post-OAuth page (#110) * fix(dd-account-setup): OAuth callback port fallback, clearer .env password, clean post-OAuth page - Auto-select a free loopback callback port when 8080 is busy (RFC 8252), threading it through the authorize URL, the statefile, and the token-exchange redirect_uri; prefer 8080 so the common path is unchanged. - Surface the generated Path C password at the Step 5 credential card and in the C2/handoff wording (location + last 4 only, never the full value). - Serve the post-OAuth page as charset=utf-8 with an ASCII-only body so it can't render as mojibake. Security-sensitive logic (PKCE, state, token exchange, password generation) is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(dd-account-setup): close OAuth callback port race + scope generated-password display Addresses the two Copilot review findings on datadog-labs/agent-skills#110. 1) OAuth callback port race (authenticate.md, Step 1). The port was picked in one python process (bind -> getsockname -> close) and the listener bound in a second, so the port was released before the listener bound: another local process could take it (forcing the paste fallback, or catching the redirect in the wrong process). Merge both into ONE process that binds once, HOLDS the socket, writes ver/st/port, opens the browser, then serves one request on that same held socket -- the authorize URL now always names the port actually being listened on. (PKCE + the 0600 statefile already made an intercepted code unusable; this removes the race itself.) No-python or no-free-port still fall back to :8080 + paste. 2) Stale generated-password display (SKILL.md Step 5; authenticate.md C3). The Step 5 card grepped .env unconditionally, so a later Path A/B run surfaced a leftover DD_SIGNUP_PASSWORD from an earlier signup, and `head -1` returned the OLDEST entry after C2 appends. Gate the display on a run-scoped marker written only when THIS run creates the account (C3), and read the newest entry (tail -1). Fix C3's own read to tail -1 too, so a re-run signs up with the freshly generated password rather than a stale one. (.env is the only file C2 writes the password to, so reading .env -- not .env.local -- is correct here.) Verified offline: `bash -n` on all 12 shell blocks; the listener compiles and, run live, binds+serves a single socket with URL port == served port == statefile port on both the 8080-free and 8080-busy (ephemeral) paths, capturing the redirect and writing 0600 files; marker gating shows the newest password only when this run created the account and nothing on a later run.

Add cloud integration skills (AWS, Azure, GCP, OCI) (#101) * Add dd-aws-integration skill Co-Authored-By: Claude Opus 5 (1M context) * Add dd-azure-integration skill Co-Authored-By: Claude Opus 5 (1M context) * Add dd-gcp-integration skill Co-Authored-By: Claude Opus 5 (1M context) * Add dd-oci-integration skill Co-Authored-By: Claude Opus 5 (1M context) * Correct the UK1 trusted account and the Terraform key-passing note UK1 is a real Datadog site with its own AWS trusted account, 117348461845 (verified against DdAccountIdBySite in DataDog/cloudformation-template). Folding it into the 464622532012 row writes an sts:AssumeRole trust for the wrong account, so a UK1 install collects nothing. The keys also never travel through TF_VAR_* here and nothing is declared ephemeral: the datadog provider reads DD_API_KEY / DD_APP_KEY from the environment. Leaving the stale note in can push a model into inventing key variables the template prohibits. * Correct the Terraform key-passing note The keys never travel through TF_VAR_* and nothing is declared ephemeral: the datadog provider reads DD_API_KEY / DD_APP_KEY from the environment. The client-secret caveat below the bullet is unchanged and still applies. * Fix the google_projects folder filter and the key-passing note data.google_projects calls Cloud Resource Manager v1, whose filter runs server-side and does not share the gcloud --filter grammar used during discovery: it has no AND/OR/NOT keywords, requires parent.type alongside parent.id, and has no projectId field. The old filter therefore fails with an invalid-filter error or mis-parses, so a folder-based install enables APIs on the wrong or empty project set and silently collects nothing. The sys-* exclusion has no valid v1 form and moves client-side into all_project_ids. The keys also never travel through TF_VAR_* and nothing is declared ephemeral: the datadog provider reads DD_API_KEY / DD_APP_KEY from the environment. * Honour the selected OCI profile and pin the module revision The credential and OCID checks read OCI_CLI_PROFILE and fell back to DEFAULT, so naming a profile in conversation was not enough: the checks inspected a different identity than the one being installed. Phase 1 now settles the profile up front and the oci CLI calls pass it explicitly. The module source also carried no ref, and Terraform's lock file does not lock remote module revisions, so every fresh init resolved whatever sat on the default branch. This module creates IAM users, policies, API keys and Vault resources, so pin it: datadog-integration-v1.1.17, which is byte-identical to master today. * Note OpenTofu as a drop-in substitute for Terraform Every command in the four cloud integration skills is written as `terraform`, so an agent working with a user who only has `tofu` installed fails at the first preflight check. Add a Phase 0 note to detect the binary and substitute throughout. The provider short-form sources and the OCI module's pinned GitHub source resolve identically under both, so no other change is needed. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) Co-authored-by: Ryan MacLean

Add dd-instrument-rum skill (#106) ## Add `dd-instrument-rum` skill Adds a skill that instruments a browser web application with **Datadog Browser RUM** — detecting the framework and safely adding (or completing) classic RUM setup without breaking the build. Instrumentation only; it never adds APM, Logs, LLM Observability, source-map upload, or unrelated products. ### How it works 1. **Analyze** — locates the browser app the real build ships (monorepo-aware), then detects framework, router, package manager, bundler, entrypoint, TypeScript, and any existing RUM init. 2. **Route** — reads a shared `rum-core` reference plus exactly one framework reference (React, Next.js App/Pages Router, Angular, Vue, Nuxt, or Svelte/vanilla/SPA/iframe), copying package names, imports, and init options verbatim rather than from memory. 3. **Instrument** — installs `@datadog/browser-rum` and the matching framework package at one pinned version, initializes RUM exactly once at the earliest browser entrypoint, and wires router + framework error integration only where those surfaces already exist. 4. **Verify** — confirms a single init path, persists packages to the real manifest/lockfile, runs the project's own build, and reports a short JSON result — never claiming telemetry it didn't observe. ### Problems it solves - "Add / set up / instrument Datadog RUM in my frontend" across React, Next.js, Angular, Vue, Nuxt, Svelte, and vanilla apps. - **Repairing** a partial setup — completes missing plugin/router/error wiring while treating an existing valid init's credentials and options as immutable (no duplicate init, no credential clobbering). - The common failure modes: init in a render/hook, mixed npm+CDN, mismatched SDK versions, or a green install mistaken for a passing build. ### Dependencies - **Browser SDK packages only** — `@datadog/browser-rum` plus the relevant `@datadog/browser-rum-*` integration, installed with the project's own package manager. - Optionally uses a `CreateRumApplication` tool when present to provision a RUM app; otherwise falls back to asking for an existing Application ID / Client Token / site. No API keys — RUM uses browser-visible identifiers.

feat(agent-observability): generalize experiment bootstrap (#103) * feat(agent-observability): generalize experiment bootstrap * refactor(agent-observability): defer HTTP adapter * docs(agent-observability): restore Python bootstrap reference * refactor(agent-observability): rename experiment bootstrap skill * refactor(experiments): align adapter reference directories

Merge pull request #102 from sakibh/auto-experiment-cost-estimate auto-experiment: derive a run-cost estimate

auto-experiment: derive a run-cost estimate Adds a cost estimate to the step-3 intake recap, derived from real call counts and measured/estimated token usage (historical traces preferred, static code analysis as fallback) rather than asked of the user, who rarely knows what a single case costs. Explicitly split from orchestration cost, with anti-fabrication rules for pricing lookups.

Merge pull request #97 from sakibh/auto-experiment-explicit-domain-notes-prompt auto-experiment: ask domain_notes explicitly instead of mentioning it in the config review

auto-experiment: ask domain_notes explicitly instead of mentioning it in the config review Burying the invitation inside the all-params recap meant users routinely never noticed it was a live prompt, so domain_notes stayed empty by default. Promote it to its own AskUserQuestion, asked before the recap. Co-Authored-By: Claude Sonnet 5

Add dd-account-setup skill (#96) * Add dd-account-setup skill from DataDog/agentic-onboarding-evals * dd-account-setup: address Copilot PR #96 review - Step H (headless): load DD_* from .env/.env.local before the missing-keys check, so CI runs with creds in .env (not exported) no longer fail. - Step 5: reword "source .env" -> load by parsing (never source), matching the code and avoiding a source-secrets-file footgun. - authenticate.md: send Python interpreter version/path to $DDLOG, print a clean status line (conventions.md). - get-api-key.md: redirect raw HTTP/URL/curl-exit machinery to $DDLOG; keep the "fetching a fresh key, not reusing your ambient DD_API_KEY" transparency as a single clean line.

Merge pull request #94 from ebrevan/replay-trace-intent-tagging replay-trace: tag MCP tool calls with skill name in telemetry.intent

replay-trace: collapse intent-tagging to a single line (match eval-bootstrap style) Co-Authored-By: Claude Opus 4.8

replay-trace: match the current intent-tagging convention (inv_id + :start) The house convention evolved past #370: prefix telemetry.intent with skill:[] on every MCP call, and skill::start[] on the first call only (marks the invocation start; inv_id groups one run's calls). Aligns replay-trace with eval-bootstrap/eval-pipeline/experiment- analyzer/session-classify/trace-rca. Co-Authored-By: Claude Opus 4.8

replay-trace: tag MCP tool calls with skill name in telemetry.intent Follows DataDog/llm-obs#370: prefix every datadog-llmo MCP call's telemetry.intent with "skill:agent-observability-replay-trace — " so the server can attribute tool calls (and thus skill invocations) to this skill by exact prefix match. Applies on the MCP path only; pup-CLI calls carry no intent. Co-Authored-By: Claude Opus 4.8

Add dd-product-recommender skill from DataDog/agentic-onboarding-evals (#93) * Add dd-product-recommender skill from DataDog/agentic-onboarding-evals Recommends the right Datadog products for a codebase and/or stated goal (recommendation only — no setup/install steps). Sanitized before publishing: removed internal sales/CRM provenance language (closed-won "sold and won", "deal figures", "sales correlation", "customer engagements") and re-attributed the mappings to Datadog product capabilities and common technology patterns. Tier/confidence logic is unchanged. * Fix internal inconsistencies in dd-product-recommender reference maps Addresses Copilot review on PR #93 — three genuine contradictions: - Backend languages: "All eight" miscounted (nine rows) and over-claimed foundational status for Rust/C/C++; clarify the seven GA languages are foundational and Rust/C/C++ are situational (Preview/manual). - Product Analytics: was listed in the foundational frontend bundle but is niche/explicit-match-only in the catalog; demote to a situational add. - LLM Observability theme confidence "emerging" contradicted the anchor table's "defining + well-established", which gates the lead rule; set to well-established so it can head an LLM recommendation.

Merge pull request #92 from tillwf/till.wohlfarth/remove-auto-experiment-pycache chore(auto-experiment): remove committed __pycache__ artifacts