Published copy. Evidence paths refer to the repository branch insights/verifier-screen.
Verdict against the design line: FAIL. Every pass must keep at least 55 of 61 correct proposals and accept at most 23 of 48 wrong ones.
This is a blinded screen. The verifier saw the question, the saved SQL, the plan metadata, the result rows, the schema scope and the ratified definitions. It did not see the outcome label or the expected answer. A pass here is not a correctness claim and not a deployment decision.
docs/evidence/insights-goal2-2026-09-06/phase0/fixture-schema.sql (sha256 b7122dfca379). Definitions: 58 ratified entries parsed from docs/bi/knowledge.| Pass | Correct retained | Wrong accepted | Conditional precision | Clarify | Reject correct | Reject wrong | Non-verdicts | Line |
|---|---|---|---|---|---|---|---|---|
| 1 | 28/61 | 1/48 | 96.5% | 0 | 33 | 47 | 0 | FAIL |
| 2 | 27/61 | 0/48 | 100.0% | 0 | 34 | 48 | 0 | FAIL |
| 3 | 26/61 | 1/48 | 96.3% | 0 | 35 | 47 | 0 | FAIL |
| Pooled | 81/183 | 2/144 | 97.6% | 0 | 102 | 142 | 0 | pooled verdicts |
| Majority vote | 27/61 | 1/48 | 96.4% | per case |
Agreement across passes per case: unanimous 104, majority 5, split 0, incomplete 0 of 109 cases.
dependent turn index gt 0: 59 cases.
| Pass | Correct retained | Wrong accepted | Conditional precision | Clarify |
|---|---|---|---|---|
| 1 | 22/34 | 1/25 | 95.7% | 0 |
| 2 | 21/34 | 0/25 | 100.0% | 0 |
| 3 | 20/34 | 1/25 | 95.2% | 0 |
| Pooled | 63/102 | 2/75 | 96.9% | 0 |
first turn: 50 cases.
| Pass | Correct retained | Wrong accepted | Conditional precision | Clarify |
|---|---|---|---|---|
| 1 | 6/27 | 0/23 | 100.0% | 0 |
| 2 | 6/27 | 0/23 | 100.0% | 0 |
| 3 | 6/27 | 0/23 | 100.0% | 0 |
| Pooled | 18/81 | 0/69 | 100.0% | 0 |
| Family | Correct cases | Wrong cases | Correct retained | Wrong accepted | Reject correct | Reject wrong | Clarify | Non-verdicts |
|---|---|---|---|---|---|---|---|---|
| activity_deadline_before_creation | 5 | 0 | 7 | 0 | 8 | 0 | 0 | 0 |
| activity_duration_missing_due_time | 5 | 0 | 6 | 0 | 9 | 0 | 0 | 0 |
| all_commission_unknown | 0 | 1 | 0 | 0 | 0 | 3 | 0 | 0 |
| booked_without_settled_history | 0 | 2 | 0 | 0 | 0 | 6 | 0 | 0 |
| campaign_and_web_referral | 3 | 0 | 9 | 0 | 0 | 0 | 0 | 0 |
| corporate_registration_without_trade | 3 | 0 | 6 | 0 | 3 | 0 | 0 | 0 |
| customer_referrals_with_reward_event | 5 | 0 | 11 | 0 | 4 | 0 | 0 | 0 |
| deal_assignment_override | 3 | 2 | 0 | 0 | 9 | 6 | 0 | 0 |
| deal_person_organization_conflict | 1 | 0 | 0 | 0 | 3 | 0 | 0 | 0 |
| deal_stage_timestamp_order | 2 | 0 | 1 | 0 | 5 | 0 | 0 | 0 |
| deleted_booking_live_account | 0 | 2 | 0 | 0 | 0 | 6 | 0 | 0 |
| first_booked_before_first_broker | 0 | 1 | 0 | 0 | 0 | 3 | 0 | 0 |
| funnel_incomplete_contact | 0 | 5 | 0 | 0 | 0 | 15 | 0 | 0 |
| fx_missing_on_settled | 0 | 2 | 0 | 0 | 0 | 6 | 0 | 0 |
| invoice_before_settlement | 0 | 2 | 0 | 0 | 0 | 6 | 0 | 0 |
| known_external_person_qualification | 0 | 5 | 0 | 2 | 0 | 13 | 0 | 0 |
| known_zero_booked_commission | 2 | 0 | 0 | 0 | 6 | 0 | 0 | 0 |
| multi_broker_activation | 0 | 1 | 0 | 0 | 0 | 3 | 0 | 0 |
| negative_realized_commission | 2 | 0 | 3 | 0 | 3 | 0 | 0 | 0 |
| notified_without_verification | 0 | 2 | 0 | 0 | 0 | 6 | 0 | 0 |
| org_contact_without_website | 2 | 0 | 0 | 0 | 6 | 0 | 0 | 0 |
| organization_client_type_multiple | 0 | 1 | 0 | 0 | 0 | 3 | 0 | 0 |
| organization_employee_data_gap | 1 | 0 | 0 | 0 | 3 | 0 | 0 | 0 |
| organization_erp_multiple | 0 | 1 | 0 | 0 | 0 | 3 | 0 | 0 |
| organization_introducer_link | 1 | 0 | 0 | 0 | 3 | 0 | 0 | 0 |
| organization_wallet_share_unknown | 2 | 0 | 0 | 0 | 6 | 0 | 0 | 0 |
| outdated_still_activated | 0 | 1 | 0 | 0 | 0 | 3 | 0 | 0 |
| overdue_expected_close | 2 | 0 | 0 | 0 | 6 | 0 | 0 | 0 |
| partner_paid_and_unpaid_statements | 5 | 0 | 15 | 0 | 0 | 0 | 0 | 0 |
| person_communication_preferences | 3 | 1 | 2 | 0 | 7 | 3 | 0 | 0 |
| person_multi_frequency_declarations | 0 | 1 | 0 | 0 | 0 | 3 | 0 | 0 |
| person_note_pinned_exclusively | 0 | 2 | 0 | 0 | 0 | 6 | 0 | 0 |
| person_organization_owner_gap | 2 | 0 | 0 | 0 | 6 | 0 | 0 | 0 |
| prebillable_eligible_stats | 0 | 2 | 0 | 0 | 0 | 6 | 0 | 0 |
| private_declared_currency_overlap | 0 | 1 | 0 | 0 | 0 | 3 | 0 | 0 |
| professional_zero_rate_referrals | 0 | 5 | 0 | 0 | 0 | 15 | 0 | 0 |
| qualified_without_deal_stage | 2 | 0 | 0 | 0 | 6 | 0 | 0 | 0 |
| quote_conversion_valid_link | 0 | 3 | 0 | 0 | 0 | 9 | 0 | 0 |
| quote_days_before_activation | 2 | 0 | 0 | 0 | 6 | 0 | 0 | 0 |
| quote_only_system_activity | 0 | 2 | 0 | 0 | 0 | 6 | 0 | 0 |
| trade_sell_not_declared | 0 | 2 | 0 | 0 | 0 | 6 | 0 | 0 |
| unknown_sti_with_broker | 0 | 1 | 0 | 0 | 0 | 3 | 0 | 0 |
| verified_after_registration | 5 | 0 | 15 | 0 | 0 | 0 | 0 | 0 |
| verified_private_rejected_registration | 3 | 0 | 6 | 0 | 3 | 0 | 0 | 0 |
Tags on reject verdicts, counted once per verdict. The second count shows the same tag on the other kind of rejection.
Rejecting wrong proposals
| Tag | On rejected wrong | Also on rejected correct |
|---|---|---|
| output-columns | 93 | 8 |
| merge-semantics | 63 | 3 |
| membership | 60 | 36 |
| predicate-missing | 53 | 39 |
| population | 43 | 41 |
| prior-turn-mismatch | 38 | 18 |
| forced-empty | 25 | 3 |
| predicate-extra | 24 | 15 |
| result-sql-mismatch | 19 | 4 |
| cohort-rederived | 12 | 23 |
| source-choice | 10 | 0 |
| discarded-source | 9 | 0 |
| identifier-mapping | 7 | 25 |
| ordering-ok | 7 | 7 |
| ordering | 7 | 0 |
Rejecting correct proposals
| Tag | On rejected correct | Also on rejected wrong |
|---|---|---|
| population | 41 | 43 |
| predicate-missing | 39 | 53 |
| membership | 36 | 60 |
| identifier-mapping | 25 | 7 |
| cohort-rederived | 23 | 12 |
| prior-turn-mismatch | 18 | 38 |
| predicate-extra | 15 | 24 |
| distinct | 9 | 5 |
| definition-conflict | 8 | 0 |
| output-columns | 8 | 93 |
| identity-mapping | 7 | 0 |
| columns-ok | 7 | 0 |
| ordering-ok | 7 | 7 |
| count-grain | 6 | 0 |
| predicate-changed | 6 | 0 |
Classes are matched on the verifier's reason sentence with fixed patterns. A verdict can fall in several classes. The primary class is the first match in class order on the first rejecting pass.
| Class | Rejected-correct verdicts | Correct cases, primary class | Rejected-wrong verdicts | Wrong cases, primary class |
|---|---|---|---|---|
| staff-exclusion | 37 | 13 | 9 | 3 |
| identifier-mapping | 34 | 1 | 10 | 2 |
| predicate-missing | 34 | 0 | 52 | 14 |
| cohort-rederived | 24 | 8 | 13 | 5 |
| house-exclusion | 14 | 4 | 3 | 1 |
| row-limit | 12 | 0 | 8 | 1 |
| discarded-source | 11 | 0 | 34 | 5 |
| count-grain | 10 | 3 | 3 | 0 |
| blocked-prior | 6 | 2 | 0 | 0 |
| output-columns | 5 | 0 | 61 | 6 |
| predicate-changed | 4 | 1 | 0 | 0 |
| forced-empty | 2 | 1 | 16 | 4 |
| other | 0 | 1 | 0 | 6 |
Majority-rejected cases: 34 correct, 47 wrong.
Reading of the correct-case classes, from the reasons and the packets:
pd-person-table-key (docs/bi/knowledge/pipedrive-activities.md) says to exclude staff with pd_is_ct_employee IS NOT TRUE. The packet supplied it for every question that names Pipedrive people. The holdout questions and their sealed answers do not apply that exclusion. The verifier sided with the definition. This is the design's shared-misunderstanding risk in reverse: a supplied definition vetoed correct answers.| Case | Turn | Label | Verdicts | Majority | Reason (pass 1) |
|---|---|---|---|---|---|
| verified_after_registration:first-count | 0 | correct | accept / accept / accept | accept | The SQL counts distinct CT accounts of any role whose earliest non-null broker verification is after completed registration and returns client_count as zero whe |
| corporate_registration_without_trade:first-count | 0 | correct | reject / reject / reject | reject | The SQL counts distinct users (c.user_id) instead of distinct client records (c.id), collapsing qualifying records that share a user. |
| quote_conversion_valid_link:first-count | 0 | wrong | reject / reject / reject | reject | The query omits the required quotes.type = 'by_client' filter and improperly excludes account 1 and accounts with CurrencyTransfer email addresses from the requ |
| quote_days_before_activation:first-count | 0 | correct | reject / reject / reject | reject | The SQL excludes account ID 1 and accounts with CurrencyTransfer email addresses, although the question requests all qualifying CT accounts of any role without |
| negative_realized_commission:first-count | 0 | correct | reject / reject / reject | reject | The SQL excludes account ID 1 and accounts with CurrencyTransfer email addresses, but the question requests CT accounts of any role without those exclusions. |
| known_zero_booked_commission:first-count | 0 | correct | reject / reject / reject | reject | The SQL excludes account ID 1 and accounts with CurrencyTransfer email addresses, adding an unsupported restriction to the requested population of CT accounts o |
| professional_zero_rate_referrals:first-count | 0 | wrong | reject / reject / reject | reject | The query forces zero with AND FALSE, omits the zero commission-rate predicate, and counts distinct user IDs rather than distinct client record IDs. |
| customer_referrals_with_reward_event:first-count | 0 | correct | reject / reject / reject | reject | The query counts distinct account IDs (c.user_id) rather than distinct client records (c.id), collapsing multiple client records belonging to the same account. |
| partner_paid_and_unpaid_statements:first-count | 0 | correct | accept / accept / accept | accept | The query counts each professional partner once when it has both paid and unpaid monthly statements and returns client_count as zero when none qualify. |
| campaign_and_web_referral:first-count | 0 | correct | accept / accept / accept | accept | The SQL counts distinct CT accounts of any role with a non-null campaign assignment and a nonblank web referral domain, returning client_count with zero when no |
| funnel_incomplete_contact:first-count | 0 | wrong | reject / reject / reject | reject | The SQL returns an unrequested phone column alongside client_count, contradicting both the requested output and the reported result columns. |
| verified_private_rejected_registration:first-count | 0 | correct | reject / reject / reject | reject | The SQL counts distinct users (c.user_id) rather than distinct client records (c.id), collapsing multiple qualifying client records belonging to one user. |
| activity_deadline_before_creation:first-count | 0 | correct | reject / reject / reject | reject | The query omits the required staff exclusion using persons_enriched.pd_is_ct_employee IS NOT TRUE, so client_count can include CT employees. |
| person_communication_preferences:first-count | 0 | wrong | reject / reject / reject | reject | The SQL forces an empty result instead of counting distinct Pipedrive people whose preference array includes email and excludes phone, fails to return zero when |
| deal_assignment_override:first-count | 0 | correct | reject / reject / reject | reject | The query omits the required CT employee exclusion and uses the activity rep as a fallback assignee without a definition authorizing that substitution. |
| known_external_person_qualification:first-count | 0 | wrong | reject / reject / reject | reject | The query returns a constant zero without counting distinct Pipedrive people or checking explicit non-employee status and qualification after signup. |
| activity_duration_missing_due_time:first-count | 0 | correct | reject / reject / reject | reject | The query omits the required staff exclusion via persons_enriched.pd_is_ct_employee IS NOT TRUE, so client_count may include employees. |
| verified_after_registration:t1 | 0 | correct | accept / accept / accept | accept | The SQL returns distinct CT account IDs across all roles whose earliest broker verification follows completed registration, ordered ascending; Pipedrive-only pe |
| verified_after_registration:t2 | 1 | correct | accept / accept / accept | accept | The SQL preserves the prior result's CT account membership across all roles and returns its distinct count as client_count, correctly yielding zero for the empt |
| verified_after_registration:t3 | 2 | correct | accept / accept / accept | accept | The SQL preserves the prior distinct CT account cohort and predicates, returns only user_id in descending order without limiting membership, and produces the sa |
| verified_after_registration:t4 | 3 | correct | accept / accept / accept | accept | The SQL preserves the third result’s membership across all account roles, counts distinct IDs as client_count, and correctly returns zero for the empty cohort. |
| notified_without_verification:t1 | 0 | wrong | reject / reject / reject | reject | The forced-empty Gold query is the preserved merge side, so all 11 CT rows are discarded and the result fails to list the requested accounts. |
| notified_without_verification:t3 | 2 | wrong | reject / reject / reject | reject | The merge preserves 45 unfiltered Gold IDs, including 34 without qualifying CT matches, contradicts the prior empty membership, and returns client and client_ty |
| outdated_still_activated:t1 | 0 | wrong | reject / reject / reject | reject | The merge retains 35 Gold users without a qualifying CT match, omits qualifying CT users absent from Gold, and returns client and client_type despite the user_i |
| multi_broker_activation:t3 | 2 | wrong | reject / reject / reject | reject | The Gold-preserving merge includes 15 unmatched IDs and omits 18 qualifying CT-source IDs, returns columns beyond user_id, and references a checked cohort absen |
| private_declared_currency_overlap:t1 | 0 | wrong | reject / reject / reject | reject | The merge preserves 45 unfiltered Gold rows despite zero qualifying CT matches, including corporate and unknown clients, and returns extra columns instead of on |
| corporate_registration_without_trade:t2 | 1 | correct | accept / accept / accept | accept | The SQL preserves the prior cohort’s membership, counts distinct user IDs as client_count, naturally returns zero for an empty cohort, and matches the prior res |
| corporate_registration_without_trade:t4 | 3 | correct | accept / accept / accept | accept | The query preserves the prior cohort, counts distinct user IDs as client_count, and returns zero for an empty cohort. |
| quote_only_system_activity:t1 | 0 | wrong | reject / reject / reject | reject | The merge preserves 45 unfiltered Gold rows despite zero CT quote-cohort matches, and the result includes extra columns and is not ordered by user_id ascending. |
| quote_only_system_activity:t3 | 2 | wrong | reject / reject / reject | reject | The proposal re-derives the cohort with extra account exclusions and narrower quote types, preserves 45 unfiltered Gold rows despite zero CT matches, and return |
| unknown_sti_with_broker:t3 | 2 | wrong | reject / reject / reject | reject | The Gold-preserving merge omits one of seven qualifying CT IDs, returns extra columns despite the user_id-only request, and cannot establish continuity with the |
| quote_conversion_valid_link:t1 | 0 | wrong | reject / reject / reject | reject | The merge preserves the forced-empty Gold source and discards all 60 CT rows, while the CT query also omits the required quotes.type = 'by_client' predicate. |
| quote_days_before_activation:t1 | 0 | correct | reject / reject / reject | reject | The merge preserves a Gold query forced empty by HAVING FALSE, while the CT query adds unauthorized ID/email exclusions and a 10,001-row limit despite requestin |
| quote_conversion_valid_link:t3 | 2 | wrong | reject / reject / reject | reject | The Gold-preserving merge retains 4 unqualified IDs and omits 19 CT-qualified IDs, changes the prior result's membership, and returns extra columns despite the |
| booked_without_settled_history:t1 | 0 | wrong | reject / reject / reject | reject | The merge retains 41 unmatched Gold users outside the qualifying CT cohort, the CT query excludes accounts despite “any role,” and the result includes columns b |
| invoice_before_settlement:t1 | 0 | wrong | reject / reject / reject | reject | The Gold-preserving merge includes four accounts without qualifying bookings, omits 19 qualifying CT accounts, and returns client and client_type despite the re |
| booked_without_settled_history:t3 | 2 | wrong | reject / reject / reject | reject | The result includes unrequested columns, re-derives the prior cohort, and preserves 41 Gold IDs without a qualifying CT match. |
| prebillable_eligible_stats:t1 | 0 | wrong | reject / reject / reject | reject | The Gold-preserving merge retains unmatched accounts and drops CT-only accounts, while the SQL adds unrequested account exclusions, omits the deleted-booking fi |
| invoice_before_settlement:t3 | 2 | wrong | reject / reject / reject | reject | The result includes client and client_type despite the user_id-only requirement, and the Gold-preserving merge retains four IDs without qualifying CT matches wh |
| prebillable_eligible_stats:t3 | 2 | wrong | reject / reject / reject | reject | The result includes client and client_type despite the user_id-only request, and the Gold-preserving merge retains eight IDs without qualifying CT matches. |
| deleted_booking_live_account:t1 | 0 | wrong | reject / reject / reject | reject | The SQL substitutes closed status for booking deletion, omits the nondeleted-account filter, restricts membership to Gold accounts, and the result includes unre |
| fx_missing_on_settled:t1 | 0 | wrong | reject / reject / reject | reject | The merge retains 41 users without a qualifying CT booking and returns client and client_type columns despite the request for only user_id. |
| deleted_booking_live_account:t3 | 2 | wrong | reject / reject / reject | reject | The result includes unrequested client and client_type columns and retains 39 Gold IDs unmatched by the CT eligibility check, while re-deriving membership inste |
| fx_missing_on_settled:t3 | 2 | wrong | reject / reject / reject | reject | The result includes client and client_type despite the explicit requirement to show only user_id. |
| negative_realized_commission:t2 | 1 | correct | accept / accept / accept | accept | The query preserves the first result’s membership and counts its distinct grouped user IDs as client_count, returning zero for an empty cohort and matching the |
| all_commission_unknown:t3 | 2 | wrong | reject / reject / reject | reject | The merge retains 41 unmatched Gold IDs, drops one qualifying CT ID, returns extra columns despite the user_id-only requirement, and references a checked cohort |
| known_zero_booked_commission:t2 | 1 | correct | reject / reject / reject | reject | The SQL changes the first result’s membership by omitting the deleted-booking exclusion and adding exclusions for account ID 1 and CurrencyTransfer email addres |
| first_booked_before_first_broker:t3 | 2 | wrong | reject / reject / reject | reject | The merge retains 27 unmatched Gold IDs and drops 9 CT cohort IDs, the CT query adds unrequested account exclusions, the output includes extra columns, and the |
| customer_referrals_with_reward_event:t1 | 0 | correct | accept / accept / accept | accept | The query returns distinct CT client user IDs whose referrer has role 'User' and a reward event for that same client, ordered ascending, and the empty result is |
| professional_zero_rate_referrals:t1 | 0 | wrong | reject / reject / reject | reject | The SQL omits the referrer's zero commission-rate filter, and the result includes client and client_type although both the request and SQL specify only user_id. |
| customer_referrals_with_reward_event:t2 | 1 | correct | accept / accept / accept | accept | The SQL preserves the prior cohort’s customer-referrer and same-client reward predicates, counts distinct user IDs as client_count, and returns zero for the emp |
| professional_zero_rate_referrals:t2 | 1 | wrong | reject / reject / reject | reject | The query omits the prior result’s public.clients join, so it counts professionally referred accounts without preserving the requested CT client membership. |
| customer_referrals_with_reward_event:t3 | 2 | correct | accept / accept / reject | accept | The query retains the prior referral and same-client reward predicates, returns only distinct user_id values in descending order, and preserves the prior empty |
| customer_referrals_with_reward_event:t4 | 3 | correct | accept / accept / accept | accept | The SQL preserves the prior cohort by matching each customer referrer’s reward event to the same client and counts distinct user IDs as client_count, returning |
| professional_zero_rate_referrals:t3 | 2 | wrong | reject / reject / reject | reject | The SQL drops the prior client-record join and omits the zero-commission-rate filter, while the result returns client and client_type despite the explicit user_ |
| partner_paid_and_unpaid_statements:t1 | 0 | correct | accept / accept / accept | accept | The query returns only distinct professional CT partner account IDs with both paid and unpaid statements in ascending order, and Pipedrive-only people cannot sa |
| professional_zero_rate_referrals:t4 | 3 | wrong | reject / reject / reject | reject | The query omits the requested zero commission-rate filter and adds an inner clients join that can exclude IDs from the prior result’s membership. |
| partner_paid_and_unpaid_statements:t2 | 1 | correct | accept / accept / accept | accept | The query preserves the prior result’s professional partner membership, returns its distinct count of 3 as client_count, and yields zero for an empty cohort. |
| partner_paid_and_unpaid_statements:t3 | 2 | correct | accept / accept / accept | accept | The query preserves all three previously checked professional partner IDs, applies the same paid-and-unpaid statement criteria, and returns only user_id in desc |
| partner_paid_and_unpaid_statements:t4 | 3 | correct | accept / accept / accept | accept | The query preserves the prior result’s professional partner membership with both paid and unpaid statements and counts distinct IDs as client_count, returning z |
| campaign_and_web_referral:t2 | 1 | correct | accept / accept / accept | accept | The SQL preserves the prior CT account cohort with identical campaign and known-domain predicates, counts distinct accounts as client_count, and returns zero fo |
| funnel_incomplete_contact:t1 | 0 | wrong | reject / reject / reject | reject | The SQL selects phone and the result includes client, client_type, and phone, violating the explicit requirement to return only user_id. |
| funnel_incomplete_contact:t2 | 1 | wrong | reject / reject / reject | reject | The SQL also returns an unrequested phone column, contradicting the required client_count-only output and the reported result columns. |
| campaign_and_web_referral:t4 | 3 | correct | accept / accept / accept | accept | The SQL preserves the prior cohort across all account roles, counts distinct IDs as client_count, and returns zero for an empty cohort. |
| trade_sell_not_declared:t1 | 0 | wrong | reject / reject / reject | reject | The merge preserves the Gold side, whose WHERE FALSE forces zero rows, discarding all 60 qualifying CT results and returning an empty list. |
| trade_sell_not_declared:t3 | 2 | wrong | reject / reject / reject | reject | The merge replaces the prior empty result with 45 Gold IDs, includes four unmatched IDs, omits 19 qualifying CT IDs, and returns extra columns despite the user_ |
| funnel_incomplete_contact:t3 | 2 | wrong | reject / reject / reject | reject | All 15 prior IDs appear in descending order, but the SQL projects phone and the result adds client, client_type, and phone despite the explicit request for only |
| funnel_incomplete_contact:t4 | 3 | wrong | reject / reject / reject | reject | The SQL projects an unrequested phone column alongside client_count, contradicting the reported single-column result. |
| person_organization_owner_gap:t1 | 0 | correct | reject / reject / reject | reject | The SQL always returns pe.id as user_id instead of using ct_id for linked people and falling back to the person ID only for Pipedrive-only people. |
| verified_private_rejected_registration:t2 | 1 | correct | accept / accept / accept | accept | The query preserves the prior cohort’s predicates, returns its distinct user count as client_count, and yields zero when empty, with the result matching the two |
| person_organization_owner_gap:t3 | 2 | correct | reject / reject / reject | reject | The SQL re-derives the owner-mismatch cohort instead of preserving the checked IDs from the first result, violating the required membership preservation. |
| verified_private_rejected_registration:t4 | 3 | correct | accept / accept / accept | accept | The SQL preserves the prior result’s private, registration-rejected, broker-verified membership and counts distinct user IDs as client_count, returning zero for |
| deal_person_organization_conflict:t1 | 0 | correct | reject / reject / reject | reject | The SQL always returns the person ID instead of using the CT user ID when available, omits the required staff exclusion, and limits a request for every match to |
| qualified_without_deal_stage:t1 | 0 | correct | reject / reject / reject | reject | The query returns person IDs even for CT-linked people, omits the required staff exclusion, matches stages beyond Negotiation, and caps an exhaustive listing at |
| qualified_without_deal_stage:t3 | 2 | correct | reject / reject / reject | reject | The SQL re-derives the cohort and applies a 10,000-row cap instead of preserving and reordering every checked ID from the first result. |
| person_note_pinned_exclusively:t1 | 0 | wrong | reject / reject / reject | reject | The query omits the person/deal pin predicates and staff exclusion, incorrectly requires deal_id IS NULL, fails to use CT user IDs for linked people, and impose |
| activity_deadline_before_creation:t1 | 0 | correct | reject / reject / reject | reject | The query returns person IDs for everyone instead of CT user IDs with a person-ID fallback for Pipedrive-only people, and omits the required staff exclusion. |
| person_note_pinned_exclusively:t3 | 2 | wrong | reject / reject / reject | reject | The SQL re-derives the checked cohort, substitutes person/deal associations for the required pinning predicates, and imposes a 10,000-row cap despite the reques |
| activity_deadline_before_creation:t2 | 1 | correct | accept / accept / accept | accept | The SQL preserves the first result's distinct person membership, returns its count of 54 as client_count, and yields zero if the cohort is empty. |
| overdue_expected_close:t1 | 0 | correct | reject / reject / reject | reject | The query returns person IDs even for CT-linked people instead of their user IDs and omits the defined exclusions for archived deals and staff. |
| activity_deadline_before_creation:t3 | 2 | correct | accept / reject / reject | reject | The SQL preserves the prior distinct-person selection and returns all 54 members with only user_id, ordered descending. |
| overdue_expected_close:t3 | 2 | correct | reject / reject / reject | reject | The SQL re-derives membership from live tables instead of preserving the checked IDs from the first result as required. |
| activity_deadline_before_creation:t4 | 3 | correct | accept / accept / accept | accept | The query preserves the third result’s membership, counts its 54 distinct person IDs as client_count, and returns zero for an empty cohort. |
| organization_introducer_link:t1 | 0 | correct | reject / reject / reject | reject | The query always returns person IDs instead of using CT user IDs when available, omits the required staff exclusion, and permits self-introducing organizations. |
| organization_employee_data_gap:t1 | 0 | correct | reject / reject / reject | reject | The query uses the person ID even for CT-linked people and omits the required staff exclusion. |
| person_communication_preferences:t2 | 1 | correct | reject / reject / reject | reject | The blocked prior turn established no checked membership to preserve, and the SQL adds unrequested email and phone columns absent from the reported result. |
| person_multi_frequency_declarations:t3 | 2 | wrong | reject / reject / reject | reject | The merge retains 45 unfiltered Gold clients despite zero qualifying CT matches, fails to establish the requested Pipedrive cohort, returns extra columns, and c |
| person_communication_preferences:t3 | 2 | correct | reject / reject / reject | reject | The SQL adds email and phone columns despite the user requesting only user_id, and it cannot preserve checked membership from a prior turn that was blocked and |
| person_communication_preferences:t4 | 3 | correct | accept / reject / accept | accept | The query preserves the third result's email-but-not-phone membership and returns 32 distinct IDs as client_count, with COUNT returning zero for an empty cohort |
| org_contact_without_website:t1 | 0 | correct | reject / reject / reject | reject | The query uses person IDs for all people instead of CT user IDs for linked people and person IDs only for Pipedrive-only people, and it omits the required staff |
| deal_assignment_override:t1 | 0 | wrong | reject / reject / reject | reject | The SQL compares the assignee to the activity’s user_id instead of the deal’s owner_id and returns person IDs even when a CT user ID exists. |
| org_contact_without_website:t3 | 2 | correct | reject / reject / reject | reject | The SQL re-derives the cohort from the underlying tables instead of preserving the first result’s checked IDs, contrary to the required membership preservation. |
| deal_assignment_override:t2 | 1 | correct | reject / reject / reject | reject | The SQL changes the comparison from the activity’s user_id to the deal’s owner_id, returning 43 instead of preserving the first result’s empty membership and re |
| deal_assignment_override:t3 | 2 | wrong | reject / reject / reject | reject | The SQL compares the assignee with a.user_id instead of d.owner_id, recomputes the prior cohort, and caps membership at 10000 despite the request to keep every |
| deal_assignment_override:t4 | 3 | correct | reject / reject / reject | reject | The proposal changes the prior membership predicate and returns 43, whereas preserving the third result's empty membership requires client_count to be 0. |
| known_external_person_qualification:t1 | 0 | wrong | reject / reject / reject | reject | The query forces an empty result without reading Pipedrive people, applying the explicit non-employee and qualification-after-signup predicates, or implementing |
| known_external_person_qualification:t2 | 1 | wrong | accept / reject / accept | accept | The explicitly preserved prior result has no members, so the single client_count value of 0 correctly answers this follow-up. |
| known_external_person_qualification:t3 | 2 | wrong | reject / reject / reject | reject | The SQL lists CT accounts without the required Pipedrive predicates or preserving prior membership, and the result includes extra columns absent from both the r |
| organization_erp_multiple:t1 | 0 | wrong | reject / reject / reject | reject | The SQL forces an empty result without identifying Pipedrive people whose organizations name at least two ERP systems. |
| known_external_person_qualification:t4 | 3 | wrong | reject / reject / reject | reject | WHERE FALSE forces a zero count instead of preserving and counting the prior result's membership, and the query never checks the requested Pipedrive predicates. |
| activity_duration_missing_due_time:t1 | 0 | correct | reject / reject / reject | reject | The query uses person IDs for everyone instead of CT user IDs with person-ID fallback for Pipedrive-only people, and it omits the required staff exclusion. |
| organization_wallet_share_unknown:t1 | 0 | correct | reject / reject / reject | reject | The SQL uses the person ID for everyone instead of only Pipedrive-only people and omits the required staff exclusion. |
| activity_duration_missing_due_time:t2 | 1 | correct | accept / accept / accept | accept | The SQL preserves the first result’s membership predicates, returns its 22 distinct people as client_count, and naturally returns zero for an empty cohort. |
| organization_wallet_share_unknown:t3 | 2 | correct | reject / reject / reject | reject | The SQL re-derives the cohort from current tables instead of preserving the first result’s checked IDs, violating the requirement to keep exactly the same membe |
| activity_duration_missing_due_time:t3 | 2 | correct | reject / reject / reject | reject | The SQL re-derives membership from activities instead of preserving the checked IDs from the first result, contrary to the required cohort preservation. |
| deal_stage_timestamp_order:t1 | 0 | correct | reject / reject / reject | reject | The SQL returns person IDs for CT-linked people instead of their user_id and omits the required staff exclusion. |
| activity_duration_missing_due_time:t4 | 3 | correct | accept / accept / accept | accept | The SQL preserves the third result’s membership, counts its 22 distinct person IDs as client_count, and returns zero for an empty cohort. |
| organization_client_type_multiple:t1 | 0 | wrong | reject / reject / reject | reject | The SQL filters organizations by multiple distinct person pipeline statuses instead of multiple client-type tags in organizations.cf_client_type_tags. |
| deal_stage_timestamp_order:t3 | 2 | correct | reject / accept / reject | reject | The SQL re-derives membership from the base tables instead of preserving the first result’s checked IDs, contrary to the cohort-preservation definition. |
Verdict against the design line: FAIL. Verdict at the secondary threshold (at least 50 of 61 correct retained, same wrong-acceptance cap; secondary, not the design line): FAIL.
V2 tests whether the v1 over-rejection was a definition conflict or a verifier limit. Same 109 packets, byte-identical to v1 (manifest sha256 checked at load, leak guard re-run on every packet). Same model, reasoning effort, deadline and concurrency. Two passes in a fresh shuffled order. The only change is the verifier instruction, which gains these rules verbatim:
One clarification was added so the record can be counted: When you record that you would have applied such a default, use the signal tag "would-apply-default" and add a second tag naming the default, for example "default-staff-exclusion", "default-house-account-exclusion" or "default-email-domain-exclusion".
docs/evidence/insights-verifier-screen-2026-09-07 (manifest sha256 530fc0ffcc62, 109 packets). Pass seeds: 20260918, 20260919. Concurrency 4, deadline 90.0 s.| Pass | Correct retained | Wrong accepted | Conditional precision | Clarify | Reject correct | Reject wrong | Non-verdicts | Line at 55 | Secondary line at 50 |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 37/61 | 6/48 | 86.1% | 0 | 24 | 42 | 0 | FAIL | FAIL (secondary) |
| 2 | 37/61 | 7/48 | 84.1% | 0 | 24 | 41 | 0 | FAIL | FAIL (secondary) |
| Pooled | 74/122 | 13/96 | 85.1% | 0 | 48 | 83 | 0 | pooled verdicts | |
| Majority vote | 37/61 | 6/48 | 86.0% | per case |
Agreement across the two passes per case: unanimous 108, split 1, incomplete 0 of 109 cases. With two passes, majority means both passes agree.
A case is in a v1 rejection class when the v1 majority verdict rejected it. The class is the v1 primary class (first pattern match on the first rejecting v1 pass).
| Group | Cases | Pass 1 accepted | Pass 2 accepted | Pooled accepted | Majority accepted |
|---|---|---|---|---|---|
| Correct, in a v1 rejection class | 34 | 10/34 | 10/34 | 20/68 | 10/34 |
| Correct, not in a v1 rejection class (v1 accepted) | 27 | 27/27 | 27/27 | 54/54 | 27/27 |
| Wrong, in a v1 rejection class | 47 | 5/47 | 6/47 | 11/94 | 5/47 |
| Wrong, not in a v1 rejection class (v1 accepted) | 1 | 1/1 | 1/1 | 2/2 | 1/1 |
Per v1 class. Recovered means the v2 majority (both passes) accepts a case that v1 majority-rejected.
| Label | v1 class | Cases | Pass 1 accepted | Pass 2 accepted | Recovered (majority accept) | Still rejected (majority) | Split | Clarify verdicts | Non-verdicts |
|---|---|---|---|---|---|---|---|---|---|
| correct | v1-majority-accept | 27 | 27 | 27 | 27 | 0 | 0 | 0 | 0 |
| correct | staff-exclusion | 13 | 2 | 2 | 2 | 11 | 0 | 0 | 0 |
| correct | cohort-rederived | 8 | 8 | 8 | 8 | 0 | 0 | 0 | 0 |
| correct | house-exclusion | 4 | 0 | 0 | 0 | 4 | 0 | 0 | 0 |
| correct | count-grain | 3 | 0 | 0 | 0 | 3 | 0 | 0 | 0 |
| correct | blocked-prior | 2 | 0 | 0 | 0 | 2 | 0 | 0 | 0 |
| correct | forced-empty | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 |
| correct | identifier-mapping | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 |
| correct | other | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 |
| correct | predicate-changed | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 |
| wrong | predicate-missing | 14 | 1 | 2 | 1 | 12 | 1 | 0 | 0 |
| wrong | other | 6 | 0 | 0 | 0 | 6 | 0 | 0 | 0 |
| wrong | output-columns | 6 | 2 | 2 | 2 | 4 | 0 | 0 | 0 |
| wrong | cohort-rederived | 5 | 1 | 1 | 1 | 4 | 0 | 0 | 0 |
| wrong | discarded-source | 5 | 0 | 0 | 0 | 5 | 0 | 0 | 0 |
| wrong | forced-empty | 4 | 0 | 0 | 0 | 4 | 0 | 0 | 0 |
| wrong | staff-exclusion | 3 | 0 | 0 | 0 | 3 | 0 | 0 | 0 |
| wrong | identifier-mapping | 2 | 0 | 0 | 0 | 2 | 0 | 0 | 0 |
| wrong | house-exclusion | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 |
| wrong | row-limit | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 |
| wrong | v1-majority-accept | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 |
A verdict carries the signal when one of its tags records that the verifier would have applied a staff, house-account or e-mail-domain default that the question did not ask for. Accepted-correct verdicts with the signal are the cases where the sealed answers and the ratified defaults disagree and the verifier accepted anyway.
| Pass | Verdicts with signal | Accept correct | Accept wrong | Reject correct | Reject wrong | Clarify |
|---|---|---|---|---|---|---|
| 1 | 13 | 9 | 0 | 4 | 0 | 0 |
| 2 | 9 | 7 | 0 | 2 | 0 | 0 |
| Pooled | 22 | 16 | 0 | 6 | 0 | 0 |
| Case | Turn | Label | Verdicts | Signal in passes | Accepted with signal in passes | Tags |
|---|---|---|---|---|---|---|
| activity_deadline_before_creation:first-count | 0 | correct | accept / accept | 1 | 1 | default-staff-exclusion, would-apply-default |
| activity_duration_missing_due_time:first-count | 0 | correct | accept / accept | 1, 2 | 1, 2 | default-staff-exclusion, would-apply-default |
| person_organization_owner_gap:t3 | 2 | correct | accept / accept | 1, 2 | 1, 2 | default-staff-exclusion, would-apply-default |
| deal_person_organization_conflict:t1 | 0 | correct | reject / reject | 2 | - | default-staff-exclusion, would-apply-default |
| activity_deadline_before_creation:t1 | 0 | correct | reject / reject | 1 | - | default-staff-exclusion, would-apply-default |
| activity_deadline_before_creation:t3 | 2 | correct | accept / accept | 1, 2 | 1, 2 | default-staff-exclusion, would-apply-default |
| overdue_expected_close:t3 | 2 | correct | accept / accept | 2 | 2 | default-staff-exclusion, would-apply-default |
| activity_deadline_before_creation:t4 | 3 | correct | accept / accept | 1 | 1 | default-staff-exclusion, would-apply-default |
| organization_employee_data_gap:t1 | 0 | correct | reject / reject | 2 | - | default-staff-exclusion, would-apply-default |
| person_communication_preferences:t4 | 3 | correct | accept / accept | 1, 2 | 1, 2 | default-staff-exclusion, would-apply-default |
| org_contact_without_website:t1 | 0 | correct | reject / reject | 1 | - | default-staff-exclusion, would-apply-default |
| org_contact_without_website:t3 | 2 | correct | accept / accept | 1 | 1 | default-staff-exclusion, would-apply-default |
| activity_duration_missing_due_time:t1 | 0 | correct | reject / reject | 1 | - | default-staff-exclusion, would-apply-default |
| organization_wallet_share_unknown:t1 | 0 | correct | reject / reject | 1 | - | default-staff-exclusion, would-apply-default |
| activity_duration_missing_due_time:t2 | 1 | correct | accept / accept | 1 | 1 | default-staff-exclusion, would-apply-default |
| organization_wallet_share_unknown:t3 | 2 | correct | accept / accept | 1, 2 | 1, 2 | default-staff-exclusion, would-apply-default |
| deal_stage_timestamp_order:t3 | 2 | correct | accept / accept | 2 | 2 | default-staff-exclusion, would-apply-default |
| Class | Rejected-correct verdicts | Correct cases, primary class | Rejected-wrong verdicts | Wrong cases, primary class |
|---|---|---|---|---|
| identifier-mapping | 23 | 11 | 5 | 2 |
| house-exclusion | 9 | 4 | 2 | 1 |
| discarded-source | 6 | 0 | 15 | 3 |
| count-grain | 6 | 3 | 2 | 1 |
| blocked-prior | 4 | 2 | 0 | 0 |
| forced-empty | 2 | 0 | 10 | 4 |
| predicate-changed | 1 | 1 | 0 | 0 |
| predicate-missing | 1 | 0 | 31 | 11 |
| output-columns | 1 | 0 | 22 | 4 |
| other | 0 | 3 | 0 | 13 |
| staff-exclusion | 0 | 0 | 4 | 2 |
Majority-rejected cases in v2: 24 correct, 41 wrong.
Reading of the v1 classes under the v2 rules, from the reasons, the signals and the packets:
dependent turn index gt 0: 59 cases.
| Pass | Correct retained | Wrong accepted | Conditional precision | Clarify |
|---|---|---|---|---|
| 1 | 29/34 | 6/25 | 82.9% | 0 |
| 2 | 29/34 | 7/25 | 80.6% | 0 |
| Pooled | 58/68 | 13/50 | 81.7% | 0 |
first turn: 50 cases.
| Pass | Correct retained | Wrong accepted | Conditional precision | Clarify |
|---|---|---|---|---|
| 1 | 8/27 | 0/23 | 100.0% | 0 |
| 2 | 8/27 | 0/23 | 100.0% | 0 |
| Pooled | 16/54 | 0/46 | 100.0% | 0 |
| Case | Turn | Label | v1 majority | v1 class | v2 verdicts | v2 majority | Default signal (passes) | Reason (pass 1) |
|---|---|---|---|---|---|---|---|---|
| verified_after_registration:first-count | 0 | correct | accept | - | accept / accept | accept | - | The SQL counts distinct CT accounts of any role whose earliest non-null broker verification is after completed registration and returns client_count as zero whe |
| corporate_registration_without_trade:first-count | 0 | correct | reject | count-grain | reject / reject | reject | - | The SQL counts distinct users (c.user_id) instead of distinct client records (c.id), collapsing qualifying records that share a user. |
| quote_conversion_valid_link:first-count | 0 | wrong | reject | house-exclusion | reject / reject | reject | - | The SQL omits the required q.type = 'by_client' filter for accounts’ own quotes and improperly excludes account 1 and currencytransfer.com email accounts from t |
| quote_days_before_activation:first-count | 0 | correct | reject | house-exclusion | reject / reject | reject | - | The SQL excludes account ID 1 and accounts with @currencytransfer.com email addresses, although the question requests all qualifying CT accounts of any role wit |
| negative_realized_commission:first-count | 0 | correct | reject | house-exclusion | reject / reject | reject | - | The SQL excludes account 1 and accounts with @currencytransfer.com email addresses, although the question requests all CT accounts of any role without those exc |
| known_zero_booked_commission:first-count | 0 | correct | reject | house-exclusion | reject / reject | reject | - | The SQL excludes account 1 and accounts with @currencytransfer.com email addresses, but the question requests accounts of any role without those exclusions. |
| professional_zero_rate_referrals:first-count | 0 | wrong | reject | identifier-mapping | reject / reject | reject | - | The query forces zero with AND FALSE, omits the affiliate commission_rate = 0 predicate, and counts distinct users instead of distinct client records. |
| customer_referrals_with_reward_event:first-count | 0 | correct | reject | count-grain | reject / reject | reject | - | The SQL counts distinct c.user_id account IDs rather than c.id client record IDs, collapsing multiple client records belonging to the same account. |
| partner_paid_and_unpaid_statements:first-count | 0 | correct | accept | - | accept / accept | accept | - | The query counts distinct professional partner accounts with at least one paid and one unpaid affiliate statement and returns client_count as zero when none qua |
| campaign_and_web_referral:first-count | 0 | correct | accept | - | accept / accept | accept | - | The query counts distinct CT accounts across all roles with a campaign assignment and a nonblank web referral domain, returning client_count as zero when none q |
| funnel_incomplete_contact:first-count | 0 | wrong | reject | output-columns | reject / reject | reject | - | The SQL projects an unrequested phone column, contradicting both the requested client_count-only output and the reported result schema. |
| verified_private_rejected_registration:first-count | 0 | correct | reject | count-grain | reject / reject | reject | - | The query counts distinct c.user_id values rather than distinct client records (c.id), collapsing multiple qualifying client records belonging to the same user. |
| activity_deadline_before_creation:first-count | 0 | correct | reject | staff-exclusion | accept / accept | accept | 1 | The query counts distinct Pipedrive people with an activity due before its creation date and returns client_count as zero when none qualify. |
| person_communication_preferences:first-count | 0 | wrong | reject | output-columns | reject / reject | reject | - | The query forces an empty result instead of counting distinct Pipedrive people with email but not phone preferences and returning one client_count value, includ |
| deal_assignment_override:first-count | 0 | correct | reject | staff-exclusion | reject / reject | reject | - | Falling back to a.user_id can count activities without an identified assignee merely because their PD rep differs from the deal owner, which does not establish |
| known_external_person_qualification:first-count | 0 | wrong | reject | staff-exclusion | reject / reject | reject | - | The SQL returns a hard-coded zero without counting distinct Pipedrive people or checking explicit non-employee status and qualification after signup. |
| activity_duration_missing_due_time:first-count | 0 | correct | reject | staff-exclusion | accept / accept | accept | 1, 2 | The SQL counts distinct Pipedrive person IDs with a duration and no due time, returns client_count, and yields zero when none qualify. |
| verified_after_registration:t1 | 0 | correct | accept | - | accept / accept | accept | - | The query returns distinct CT account IDs across all roles whose earliest broker verification follows completed registration, with only user_id ordered ascendin |
| verified_after_registration:t2 | 1 | correct | accept | - | accept / accept | accept | - | The SQL preserves the prior result’s exact membership across all roles and returns its distinct account count as client_count, correctly yielding zero when empt |
| verified_after_registration:t3 | 2 | correct | accept | - | accept / accept | accept | - | The query preserves the prior cohort of distinct CT account IDs across all roles and returns only user_id in descending order, with the empty result consistent |
| verified_after_registration:t4 | 3 | correct | accept | - | accept / accept | accept | - | The query preserves the third result's membership, counts distinct account IDs as client_count, and correctly returns zero for the empty cohort. |
| notified_without_verification:t1 | 0 | wrong | reject | forced-empty | reject / reject | reject | - | The merge preserves the forced-empty Gold source, discarding all 11 qualifying CT rows and incorrectly returning no user IDs. |
| notified_without_verification:t3 | 2 | wrong | reject | discarded-source | reject / reject | reject | - | The merge retains 34 Gold IDs outside the eligible CT cohort, changes the prior result’s membership, and returns client and client_type despite the user_id-only |
| outdated_still_activated:t1 | 0 | wrong | reject | predicate-missing | reject / reject | reject | - | The merge retains 35 Gold users without qualifying CT matches, omits six qualifying CT users, and returns client and client_type despite the request for user_id |
| multi_broker_activation:t3 | 2 | wrong | reject | predicate-missing | reject / reject | reject | - | The merge retains 15 Gold IDs without a qualifying CT match, omits 18 qualifying CT IDs, and returns client and client_type despite the request for only user_id |
| private_declared_currency_overlap:t1 | 0 | wrong | reject | discarded-source | reject / reject | reject | - | The merge retains all 45 Gold rows despite zero matches to the required private-client and declared-currency predicates, and returns extra columns instead of on |
| corporate_registration_without_trade:t2 | 1 | correct | accept | - | accept / accept | accept | - | The query preserves the prior cohort’s membership, returns its distinct user count of 4 as client_count, and naturally returns zero for an empty cohort. |
| corporate_registration_without_trade:t4 | 3 | correct | accept | - | accept / accept | accept | - | The SQL preserves the prior result’s membership and returns its four distinct user IDs as client_count, with COUNT returning zero for an empty cohort. |
| quote_only_system_activity:t1 | 0 | wrong | reject | discarded-source | reject / reject | reject | - | The merge preserves 45 unfiltered Gold accounts despite an empty qualifying CT cohort, and the result includes extra columns and is not ordered by user_id ascen |
| quote_only_system_activity:t3 | 2 | wrong | reject | cohort-rederived | reject / reject | reject | - | The result violates the explicit user_id-only requirement by also returning client, client_type, and quote_count. |
| unknown_sti_with_broker:t3 | 2 | wrong | reject | predicate-missing | reject / reject | reject | - | The merge drops one of the seven qualifying CT IDs by preserving only six Gold matches, and the result includes client and client_type despite the explicit requ |
| quote_conversion_valid_link:t1 | 0 | wrong | reject | forced-empty | reject / reject | reject | - | The forced-empty Gold leg discards all 60 CT rows during the merge, and the CT query omits the required by_client quote-type filter. |
| quote_days_before_activation:t1 | 0 | correct | reject | forced-empty | reject / reject | reject | - | The preserved Gold source is forced empty by HAVING FALSE, and the CT query excludes account 1 and currencytransfer.com email addresses without authorization fr |
| quote_conversion_valid_link:t3 | 2 | wrong | reject | predicate-missing | reject / reject | reject | - | The merge retains 4 unmatched Gold IDs and omits 19 qualifying CT IDs, changes the prior empty membership, and returns client and client_type despite the reques |
| booked_without_settled_history:t1 | 0 | wrong | reject | output-columns | reject / reject | reject | - | The merge retains all 45 Gold accounts without enforcing the booking criteria, the CT query adds unrequested account and email-domain exclusions, and the result |
| invoice_before_settlement:t1 | 0 | wrong | reject | predicate-missing | reject / reject | reject | - | The Gold-preserving merge omits 19 qualifying CT users, retains 4 users without qualifying booking matches, and returns client and client_type despite the user_ |
| booked_without_settled_history:t3 | 2 | wrong | reject | cohort-rederived | reject / reject | reject | - | The result includes client, client_type, and trade_count despite the explicit request to show only user_id. |
| prebillable_eligible_stats:t1 | 0 | wrong | reject | predicate-missing | reject / reject | reject | - | The merge retains unqualified Gold accounts and drops qualifying CT accounts, the CT query adds unrequested account exclusions and omits deleted-booking filteri |
| invoice_before_settlement:t3 | 2 | wrong | reject | cohort-rederived | reject / reject | reject | - | The result includes client and client_type even though the question explicitly requests only user_id. |
| prebillable_eligible_stats:t3 | 2 | wrong | reject | other | reject / reject | reject | - | The result includes client and client_type even though the question explicitly requests only user_id. |
| deleted_booking_live_account:t1 | 0 | wrong | reject | predicate-missing | reject / reject | reject | - | The SQL filters booking status instead of requiring a deleted booking, omits the nondeleted-account predicate, excludes CT accounts absent from Gold, and return |
| fx_missing_on_settled:t1 | 0 | wrong | reject | predicate-missing | reject / reject | reject | - | The merge retains 41 unmatched Gold rows outside the qualifying CT cohort and returns client and client_type despite the request for user_id only. |
| deleted_booking_live_account:t3 | 2 | wrong | reject | cohort-rederived | reject / reject | reject | - | The result includes client and client_type even though the follow-up explicitly requests only user_id. |
| fx_missing_on_settled:t3 | 2 | wrong | reject | other | reject / reject | reject | - | The result includes client and client_type despite the explicit requirement to show only user_id. |
| negative_realized_commission:t2 | 1 | correct | accept | - | accept / accept | accept | - | The SQL preserves the first result’s exact membership and counts its distinct user IDs as client_count, returning zero if the cohort is empty. |
| all_commission_unknown:t3 | 2 | wrong | reject | output-columns | reject / reject | reject | - | The merge retains 41 Gold IDs without qualifying CT matches, omits one qualifying CT ID, and returns client and client_type despite the request for only user_id |
| known_zero_booked_commission:t2 | 1 | correct | reject | house-exclusion | reject / reject | reject | - | The SQL changes the prior cohort’s criteria by adding account-ID and email-domain exclusions and omitting the deleted-booking filter. |
| first_booked_before_first_broker:t3 | 2 | wrong | reject | discarded-source | reject / reject | reject | - | The merge keeps unqualified Gold IDs and omits unmatched qualifying CT IDs, the CT query adds unrequested account and email exclusions, and the result includes |
| customer_referrals_with_reward_event:t1 | 0 | correct | accept | - | accept / accept | accept | - | The query correctly selects distinct CT client user IDs whose customer-role referrer has a reward event for that same client, returns only user_id in ascending |
| professional_zero_rate_referrals:t1 | 0 | wrong | reject | predicate-missing | reject / reject | reject | - | The SQL omits the required zero commission-rate filter, and the result includes client and client_type despite the request for user_id only. |
| customer_referrals_with_reward_event:t2 | 1 | correct | accept | - | accept / accept | accept | - | The SQL preserves the first result's membership, correctly matches each customer referrer's reward event to the same client, and returns the distinct count as c |
| professional_zero_rate_referrals:t2 | 1 | wrong | reject | predicate-missing | reject / accept | no_majority | - | The query omits the prior result’s public.clients membership check, so it can count referred accounts outside the requested CT client records despite returning |
| customer_referrals_with_reward_event:t3 | 2 | correct | accept | - | accept / accept | accept | - | The query preserves the first result's empty membership, uses the required customer-referrer and same-client reward predicates, and returns only distinct user_i |
| customer_referrals_with_reward_event:t4 | 3 | correct | accept | - | accept / accept | accept | - | The query preserves the prior result’s membership, correctly matches each customer referrer’s reward event to the same client, and counts distinct user IDs as c |
| professional_zero_rate_referrals:t3 | 2 | wrong | reject | predicate-missing | reject / reject | reject | - | The result includes client and client_type even though both the question and SQL specify only user_id. |
| partner_paid_and_unpaid_statements:t1 | 0 | correct | accept | - | accept / accept | accept | - | The SQL returns only distinct professional CT partner user_ids with at least one paid and one unpaid monthly affiliate statement, ordered ascending. |
| professional_zero_rate_referrals:t4 | 3 | wrong | reject | predicate-missing | accept / accept | accept | - | The client join preserves the prior 32-ID membership, and COUNT(DISTINCT a.id) supplies client_count with zero for an empty set. |
| partner_paid_and_unpaid_statements:t2 | 1 | correct | accept | - | accept / accept | accept | - | The query preserves the prior result’s membership and returns the correct distinct count of 3 as client_count, with zero returned for an empty set. |
| partner_paid_and_unpaid_statements:t3 | 2 | correct | accept | - | accept / accept | accept | - | The query preserves all three previously checked partner IDs, requires both paid and unpaid statements, and returns only distinct user_id values in descending o |
| partner_paid_and_unpaid_statements:t4 | 3 | correct | accept | - | accept / accept | accept | - | The SQL preserves the prior result’s professional partner membership with both paid and unpaid statements, counts distinct IDs as client_count, and returns zero |
| campaign_and_web_referral:t2 | 1 | correct | accept | - | accept / accept | accept | - | The SQL preserves the prior cohort’s membership, counts distinct CT account IDs across all roles as client_count, and returns zero for an empty cohort. |
| funnel_incomplete_contact:t1 | 0 | wrong | reject | other | reject / reject | reject | - | The result includes client, client_type, and phone despite the explicit request to return only user_id. |
| funnel_incomplete_contact:t2 | 1 | wrong | reject | output-columns | accept / accept | accept | - | The result reports client_count as 15 for the same distinct-user cohort as the first result, and COUNT returns zero when that cohort is empty. |
| campaign_and_web_referral:t4 | 3 | correct | accept | - | accept / accept | accept | - | The query preserves the prior cohort across all account roles and correctly returns 43 distinct IDs as client_count, with zero for an empty cohort. |
| trade_sell_not_declared:t1 | 0 | wrong | reject | forced-empty | reject / reject | reject | - | The merge preserves the Gold source forced empty by WHERE FALSE, discarding all 60 qualifying CT rows and returning no user IDs. |
| trade_sell_not_declared:t3 | 2 | wrong | reject | predicate-missing | reject / reject | reject | - | The merge includes four unmatched Gold IDs, omits 19 qualifying CT IDs, contradicts the prior empty membership, and returns client and client_type despite the r |
| funnel_incomplete_contact:t3 | 2 | wrong | reject | other | reject / reject | reject | - | The result preserves all 15 prior IDs in descending order but includes client, client_type, and phone despite the explicit request to show only user_id. |
| funnel_incomplete_contact:t4 | 3 | wrong | reject | output-columns | accept / accept | accept | - | The query preserves the third result’s membership and returns its 15 distinct IDs as client_count, with COUNT yielding zero for an empty cohort. |
| person_organization_owner_gap:t1 | 0 | correct | reject | identifier-mapping | reject / reject | reject | - | The SQL returns the person ID for everyone instead of using the CT user ID for linked people and the person ID only for Pipedrive-only people. |
| verified_private_rejected_registration:t2 | 1 | correct | accept | - | accept / accept | accept | - | The SQL preserves the prior result’s membership, returns its distinct user count of 2 as client_count, and returns zero for an empty cohort. |
| person_organization_owner_gap:t3 | 2 | correct | reject | cohort-rederived | accept / accept | accept | 1, 2 | The query preserves the first result’s distinct owner-mismatch cohort, returns all 43 members with only user_id, and orders them descending. |
| verified_private_rejected_registration:t4 | 3 | correct | accept | - | accept / accept | accept | - | The query preserves the prior membership, counts distinct user IDs as client_count, and returns zero for an empty set; the reported count of 2 matches the prior |
| deal_person_organization_conflict:t1 | 0 | correct | reject | staff-exclusion | reject / reject | reject | 2 | The SQL uses the person ID for everyone instead of returning the CT user ID for linked people and using the person ID only for Pipedrive-only people. |
| qualified_without_deal_stage:t1 | 0 | correct | reject | staff-exclusion | reject / reject | reject | - | The query uses the person ID for everyone instead of returning ct_id for linked people and the person ID only for Pipedrive-only people. |
| qualified_without_deal_stage:t3 | 2 | correct | reject | cohort-rederived | accept / accept | accept | - | The query preserves the first result’s empty membership and returns only user_id with descending ordering. |
| person_note_pinned_exclusively:t1 | 0 | wrong | reject | staff-exclusion | reject / reject | reject | - | The SQL omits both required pin-flag predicates, incorrectly requires deal_id to be null, and returns person IDs without mapping CT-linked people to user_id. |
| activity_deadline_before_creation:t1 | 0 | correct | reject | staff-exclusion | reject / reject | reject | 1 | The query returns person IDs for everyone instead of CT user IDs for linked people and person IDs only for Pipedrive-only people. |
| person_note_pinned_exclusively:t3 | 2 | wrong | reject | cohort-rederived | accept / accept | accept | - | The result preserves the first result’s empty set of checked IDs, returns only user_id, and specifies descending user_id order. |
| activity_deadline_before_creation:t2 | 1 | correct | accept | - | accept / accept | accept | - | The query preserves the first result’s distinct person membership, returns its count of 54 as client_count, and returns zero for an empty set. |
| overdue_expected_close:t1 | 0 | correct | reject | staff-exclusion | reject / reject | reject | - | The SQL always returns the Pipedrive person ID as user_id, rather than using the CT user ID for linked people and the person ID only for Pipedrive-only people. |
| activity_deadline_before_creation:t3 | 2 | correct | reject | cohort-rederived | accept / accept | accept | 1, 2 | The query preserves the prior result’s 54 distinct person IDs and date predicate, returning only user_id in descending order. |
| overdue_expected_close:t3 | 2 | correct | reject | cohort-rederived | accept / accept | accept | 2 | The query preserves the first result’s empty set of checked IDs, returns only user_id, and orders by user_id descending. |
| activity_deadline_before_creation:t4 | 3 | correct | accept | - | accept / accept | accept | 1 | The query preserves the prior result’s membership, counts its 54 distinct person IDs as client_count, and returns zero for an empty set. |
| organization_introducer_link:t1 | 0 | correct | reject | staff-exclusion | reject / reject | reject | - | The SQL returns person IDs even when ct_id is available and does not require the introducer to be a different organization. |
| organization_employee_data_gap:t1 | 0 | correct | reject | staff-exclusion | reject / reject | reject | 2 | The query returns the person ID for everyone instead of using ct_id for linked people and falling back to the person ID for Pipedrive-only people. |
| person_communication_preferences:t2 | 1 | correct | reject | blocked-prior | reject / reject | reject | - | The SQL adds unrequested email and phone columns absent from the reported result, and the blocked prior turn established no membership for the count of 32 to pr |
| person_multi_frequency_declarations:t3 | 2 | wrong | reject | discarded-source | reject / reject | reject | - | The merge preserves all Gold clients despite zero matches to the multiple-frequency cohort, omits Pipedrive-person membership and the person-ID fallback, and re |
| person_communication_preferences:t3 | 2 | correct | reject | blocked-prior | reject / reject | reject | - | The prior turn was blocked and established no checked IDs to preserve, and the SQL projects email and phone despite the request for only user_id. |
| person_communication_preferences:t4 | 3 | correct | accept | - | accept / accept | accept | 1, 2 | The result counts the same 32 distinct Pipedrive person IDs using the prior membership predicates, and COUNT returns zero for an empty result. |
| org_contact_without_website:t1 | 0 | correct | reject | staff-exclusion | reject / reject | reject | 1 | The SQL uses the person ID for everyone instead of returning the linked CT user ID when available and using the person ID only for Pipedrive-only people. |
| deal_assignment_override:t1 | 0 | wrong | reject | identifier-mapping | reject / reject | reject | - | The SQL compares the assignee with the activity’s user_id instead of the deal’s owner_id and always returns person IDs rather than using ct_id for CT-linked peo |
| org_contact_without_website:t3 | 2 | correct | reject | cohort-rederived | accept / accept | accept | 1 | The query preserves the prior result’s distinct person IDs and organization filters, returning all 24 members with only user_id in descending order. |
| deal_assignment_override:t2 | 1 | correct | reject | predicate-changed | reject / reject | reject | - | The first checked result had empty membership, so preserving it requires client_count = 0, but the proposal changes the owner comparison and returns 43. |
| deal_assignment_override:t3 | 2 | wrong | reject | row-limit | accept / accept | accept | - | The result preserves the first result’s empty checked-ID set, returns only user_id, and specifies descending ordering. |
| deal_assignment_override:t4 | 3 | correct | reject | other | reject / reject | reject | - | The third result contained no IDs, so preserving its membership requires client_count = 0, but this query changes the comparison from a.user_id to d.owner_id an |
| known_external_person_qualification:t1 | 0 | wrong | reject | staff-exclusion | reject / reject | reject | - | The query forces an empty result without reading Pipedrive people or checking explicit non-employee status and qualification after signup. |
| known_external_person_qualification:t2 | 1 | wrong | accept | - | accept / accept | accept | - | The first checked result has empty membership, so client_count = 0 correctly returns its distinct count and preserves the requested scope. |
| known_external_person_qualification:t3 | 2 | wrong | reject | predicate-missing | reject / reject | reject | - | The proposal selects all CT accounts without the required Pipedrive predicates, replaces the first result’s empty membership with 70 rows, and returns extra col |
| organization_erp_multiple:t1 | 0 | wrong | reject | other | reject / reject | reject | - | The query forces an empty result without checking organizations for at least two ERP systems or selecting qualifying Pipedrive people. |
| known_external_person_qualification:t4 | 3 | wrong | reject | forced-empty | reject / reject | reject | - | WHERE FALSE forces client_count to zero instead of counting the distinct IDs in the nonempty membership from the third result. |
| activity_duration_missing_due_time:t1 | 0 | correct | reject | staff-exclusion | reject / reject | reject | 1 | The SQL returns person IDs for everyone instead of linked CT user IDs with person IDs used only for Pipedrive-only people. |
| organization_wallet_share_unknown:t1 | 0 | correct | reject | staff-exclusion | reject / reject | reject | 1 | The query always returns the person ID as user_id, omitting the linked CT user ID for people who are not Pipedrive-only. |
| activity_duration_missing_due_time:t2 | 1 | correct | accept | - | accept / accept | accept | 1 | The query preserves the prior result’s person membership, returns its distinct count of 22 as client_count, and yields zero for empty membership. |
| organization_wallet_share_unknown:t3 | 2 | correct | reject | cohort-rederived | accept / accept | accept | 1, 2 | The query preserves the prior result’s 24 distinct person IDs and returns only user_id in descending order. |
| activity_duration_missing_due_time:t3 | 2 | correct | reject | cohort-rederived | accept / accept | accept | - | The query preserves the prior result’s distinct person IDs and predicates, retains all 22 members, and returns only user_id in descending order. |
| deal_stage_timestamp_order:t1 | 0 | correct | reject | staff-exclusion | reject / reject | reject | - | The query always returns the Pipedrive person ID as user_id instead of using ct_id for linked people and the person ID only for Pipedrive-only people. |
| activity_duration_missing_due_time:t4 | 3 | correct | accept | - | accept / accept | accept | - | The query counts the same 22 distinct person IDs as the third result, preserves its predicates, and returns client_count as zero for an empty set. |
| organization_client_type_multiple:t1 | 0 | wrong | reject | other | reject / reject | reject | - | The SQL selects organizations with multiple distinct person pipeline statuses instead of organizations with more than one client-type tag in cf_client_type_tags |
| deal_stage_timestamp_order:t3 | 2 | correct | reject | cohort-rederived | accept / accept | accept | 2 | The query preserves the first result’s distinct person-ID cohort and returns all 44 members with only user_id, ordered descending. |