Experimently Documentation

Run the stack and create your first experiment.

• Quick Start—Run the stack and create your first experiment
• Core Concepts—Feature flags, experiments, variants, and metrics
• Architecture Overview—How Experimently works under the hood
• FAQ—Common questions answered
• Modules & Profiles—What the core profile includes, what each module adds, and how to run the full profile

Control feature rollouts safely with targeting rules.

• Creating Feature Flags—Set up your first flag
• Targeting & Rules—20+ operators, segment targeting
• Gradual Rollouts—Staged rollout schedules
• Safety Monitoring—Auto-rollback on error spikes

Design, run, and analyze experiments with statistical rigour.

• Running Experiments—End-to-end experiment lifecycle
• Statistical Methods—Sequential testing with mSPRT
• CUPED Variance Reduction—Beta: variance reduction; the covariate is not yet a pre-experiment metric
• Multi-Armed Bandits—Thompson Sampling, UCB1, Epsilon-Greedy
• Split URL Testing—Full profile: send each variant to a different URL. The Lambda@Edge router is a construct you deploy yourself.
• Mutual Exclusion Groups—Prevent cross-experiment contamination
• Live Results Streaming—Experiment results pushed over WebSocket every 30 seconds by default: p-values, lift and significance.

Compare prompt versions, model variants, and agent configs against real business metrics.

• Overview—What LLM evaluation is and why it matters
• Quick Start—Compare GPT-4o with Claude 3.5 Sonnet (steps 3–5 need provider API keys)

MIT-licensed client libraries for web, mobile and server.

• JavaScript SDK—Browser and Node.js
• Python SDK—Server-side Python integration
• Java SDK—Spring Boot 3 auto-configuration starter; thread-safe client with a TTL cache
• React SDK—Hooks, HOC, SSR support via ServerClient
• Go SDK—Native Go client for microservices and CLIs. Zero dependencies, context-aware, goroutine-safe.
• iOS Swift SDK—Native Swift SDK for iOS 14+ and macOS 11+. Async/await API, offline fallback.
• Android Kotlin SDK—Native Kotlin SDK for Android (minSdk 21). Coroutines, OkHttp, Compose examples.
• Flutter SDK—Dart SDK for Flutter (iOS, Android, web and desktop). Results are decided by the server, with an offline fallback.
• React Native SDK—useFlag and useExperiment hooks, Provider, AsyncStorage offline support.
• MCP tool manifest—Describes the API's tools for AI assistants.
• OpenFeature Provider—OpenFeature (a CNCF project) providers for TypeScript and Python — change flag vendors without changing app code
• Edge SDK—For Cloudflare Workers, Vercel Edge and Deno Deploy. Zero runtime dependencies; results cached in memory or KV.
• Ruby SDK—Ruby gem with zero runtime dependencies, a thread-safe TTL cache and Net::HTTP.
• PHP SDK—Composer package (ext-json and ext-curl only) with PSR-4 autoloading.
• .NET SDK—netstandard2.1 + net6.0, System.Text.Json, HttpClient, LRU cache with TTL, xUnit-tested.
• Elixir SDK—Hex package using OTP's :httpc and Jason, with a GenServer-backed ETS cache.

Full REST API documentation with request/response schemas.

• Authentication—API keys and JWT tokens
• Experiments—CRUD, scheduling, results
• Feature Flags—Flags, rollouts, evaluation
• Compliance & Audit—Full profile: signed audit events and evidence-report export
• Integrations—Jira, Salesforce, GitHub webhooks
• All Endpoints—Complete endpoint reference

Connect Experimently to your existing data stack.

• AWS—ECS Fargate, Aurora, ElastiCache, DynamoDB and Lambda
• Jira—Sync experiment status, auto-create issues
• Salesforce—OAuth2 integration, webhook sync
• GitHub—HMAC-signed webhooks, PR linking
• Slack & Email—Alerting and experiment notifications

Security, audit logging, and compliance reports.

• Compliance audit logging—Signed audit events and report export
• Compliance evidence—Full profile: evidence reports for your own audit programme
• Audit Logging—Append-only record of feature-flag status changes
• RBAC—Role-based access control: Admin, Developer, Analyst, Viewer
• API Key Management—Create, expire and delete keys; the sdk:ruleset scope for local evaluation
• SSO / SAML / OIDC—OIDC/OAuth2 for Okta, Google Workspace and GitHub. SAML 2.0 sign-in is not available yet; use OIDC.
• HIPAA controls—Full profile: PHI encryption (Fernet), a 6-year retention marker on PHI audit records, BAA records and a data-residency region check.

Run Experimently yourself: a CDK app that works with AWS, or Docker Compose on one machine.

• AWS CDK Deployment—CDK app for ECS Fargate, Aurora and ElastiCache
• Docker Compose—Local development setup
• Environment Variables—Configuration reference
• Database Migrations—Alembic migration guide
• Monitoring—CloudWatch dashboards, Prometheus metrics

Step-by-step walkthroughs for common use cases.

• Your First A/B Test—End-to-end experiment walkthrough
• Bayesian Experimentation—Beta-Binomial posteriors and stopping rules
• Guided Experiment Builder—Guided setup in the dashboard, and the wizard API
• Interaction Detection—Beta: find experiments that share users

Deep dives into the statistical methods used by the platform.

• Power Analysis & Sample Size—Pre-experiment planning: MDE, alpha, power, runtime estimation
• Post-Stratification—Variance reduction using post-experiment stratification
• FDR Correction—Benjamini-Hochberg false discovery rate correction for multiple metrics

Group members into teams, projects or product areas with workspace roles. Workspaces do not limit access to experiments or flags; that is by platform role.

• Workspace Overview—What workspaces do and do not do, workspace roles, member and API-key limits
• Quickstart—Create a workspace, invite your team, and connect an SDK