Evaluation Methodology

01The Transparency Commitment

Our business model

Our data lab is funded through affiliate partnerships. When you subscribe to a product through certain links on this site, we earn a commission from the vendor — at no extra cost to you.

Why this model cannot alter the data

An API limit is a mathematical fact. An infrastructure cost is a mathematical fact. A SOC 2 Type II certification either exists or it does not. No commission can change these realities, and our protocol is designed to make them incontestable:

• Strict separation of functions: the engineering team running audits has no visibility into partner commission rates.
• Fixed evaluation grid: every product is audited against the same published, versioned 15-point matrix. No criterion is added or removed case by case.
• Traceability: any figure we cannot verify through testing or official vendor documentation is flagged [UNVERIFIED] and excluded from scoring until validated.
• Symmetry: we document the limitations of partner products with the same rigor as products with no affiliate program.

02The Testing Protocol (The Lab)

Sandbox isolation

• Every product is tested from a fresh account with no commercial history and no privileged status with the vendor.
• No "press", "partner", or guided-demo accounts: we test what a customer pays for, at public pricing.
• Environments are destroyed and recreated on every audit cycle to neutralize residual configuration effects.

Resilience stress testing

Vendor spec sheets describe nominal behavior. Our job is to document behavior under load:

• Burst simulation: bursts of up to 10,000 API requests to observe real queue behavior, webhook handling, and recovery mechanics.
• Undocumented limit detection: published rate limits are checked against the 429 errors we actually observe. Every discrepancy is recorded on the product's audit sheet.
• Failure-path audit: auto-retry, error workflows, log persistence, and alert latency are deliberately triggered — never assumed.

03Hidden Cost Analysis (The Scale Test)

We reject teaser pricing as a basis for comparison. Our standard is true operating cost (TCO) computed for a single reference scenario: 100,000 tasks/month. The model includes:

• The pricing tier actually required for that volume — not the plan marketed most aggressively.
• Hidden billing units: the same 20-step workflow can cost 1 unit on one platform and 20 on another.
• Conditional surcharges: AI features billed at multiples, code execution, overages, options required for production use.
• For self-hosted products: infrastructure cost (compute, storage, maintenance) is included in TCO rather than presented as "free".

04Security & AI Compliance Audit

Hosting and data sovereignty

• Deployment modes actually available: multi-tenant public cloud, VPC, self-hosting (Docker, Kubernetes, on-premise, air-gapped).
• Data residency and geographic options (EU, US).

LLM API key and data transit

• Path of language-model API keys: encrypted storage, log exposure, transit through vendor intermediary servers.
• Verification of what actually leaves the customer's infrastructure when an AI agent calls an external model.

Real certifications

• We distinguish audited certifications (SOC 2 Type II, ISO 27001) from marketing language ("aligned with", "designed for compliance").
• GDPR compliance is checked at the contractual level (DPA available, subprocessors listed), not at the declarative level.
• A claimed but publicly undocumented certification is flagged [UNVERIFIED] and excluded from the security score.

05Exclusion Criteria

A product is removed from our recommendations, regardless of commercial potential, when any of the following applies:

• Severe proprietary lock-in: no usable export mechanism for workflows and data, making migration impossible without full reconstruction.
• Obsolete or misleading API documentation: documented endpoints that do not exist, published rate limits repeatedly contradicted by our measurements, no changelog.
• Structural pricing opacity: no verifiable price for our 100,000 tasks/month reference scenario without a prior sales engagement.