How to Choose a Shopify CRO Agency: 8 Questions You Must Ask Before Hiring

A founder called us in April with a proposal from another agency sitting open on his screen. Twelve pages. Nice logo. A whole section on "conversion psychology." He wanted a second opinion before signing.

We asked him one question: does the proposal mention Liquid anywhere?

He scrolled. Twice. Nothing.

That absence told us everything. The agency was going to run heatmaps and A/B test his headlines on a store that, when we opened it in DevTools two minutes later, was loading 19 third-party scripts before the hero image even started rendering. Several of those scripts belonged to apps he'd uninstalled the year before. His conversion rate was 1.4%. Nobody in that twelve-page proposal was ever going to find out why, because nobody planned to look at the code.

Every "how to choose a Shopify CRO agency" article you'll find asks you to check specialism, review case studies, and match the agency to your store's stage. Good advice. Also incomplete. None of it tells you how to find out, in a single discovery call, whether the agency you're about to pay $5,000 to $15,000 a month actually understands what's happening inside your theme, or whether they're going to spend six months testing button colors on a store that's technically broken.

The Problem: Every Vetting Guide Stops at the Business Layer

We read the top-ranking "how to choose a CRO agency" guides before writing this. They're thorough on the things that matter for picking any agency: does their client roster match your store's size, do they specialize in your platform, do they explain their testing philosophy clearly, are their case studies honest about what changed and why.

Not one of them asks whether the agency has ever traced a slow Largest Contentful Paint back to a specific dead script. Not one asks whether they test checkout on a physical iPhone or just in Chrome's device emulator. Not one asks what they'd do if your store's problem turned out to be architecture and not messaging.

That gap exists because most CRO agencies genuinely don't operate at the code level. They're behavioral specialists: heatmaps, session recordings, hypothesis-led A/B tests. That's a real discipline and it has real value once your store's rendering layer is clean. But if the actual problem is a script left over from a deleted app still executing on every page load, no amount of headline testing will move the number.

The questions below are built to expose that gap in the first conversation, before you've signed anything.

The Counter-Intuitive Insight: Ask About Failure, Not Success

Every agency you talk to will show you their best win. That tells you almost nothing, because a strong case study can come from a lucky client, a low starting baseline, or a testing period that happened to line up with a seasonal spike.

The question that actually separates a technical agency from a marketing one is this: ask them to describe a time a client's store looked fine on the surface, but the real problem turned out to be something the client, or a previous vendor, had completely missed. An agency that operates at the code level will have a specific story ready, with a mechanism, not a metric. An agency that doesn't will pivot to a generic answer about "identifying opportunities" within about two sentences.

Watch for the pivot. It's the single fastest tell in a 30-minute call.

Proof: What This Looked Like on Two Real Stores

On one engagement, a gadget brand had already tried the conventional fix themselves before calling anyone: a review widget, a popup builder, a swatch app, a sales notification tool. Thirteen apps total, each individually pitched as a conversion lift. None of it worked, because every one of those apps was injecting its own JavaScript onto the same main thread, and the underlying catalog structure was working against all of it at the same time. The fix wasn't adding a fourteenth app. It was removing the thirteen and rebuilding the variant selector and Add to Cart chain natively. LCP dropped from 6.0 seconds to under 1 second, and conversion rate moved from 1.0% to 3.3% on the same ad spend. The full numbers are in the Gadget Brand case study.

On another, a health and wellness brand had already been through a marketing agency and a design agency, both of whom tested exclusively on desktop Chrome. Neither ever picked up a physical iPhone. When we did, we found a subscription and trust-badge widget initializing after the checkout DOM had already rendered, physically pushing the Apple Pay button down the screen right as a customer's thumb completed its tap. On desktop, the shift measured 8 pixels: invisible. On a 390px iPhone viewport, the same shift pushed the button below the fold. CLS measured 0.31 against Google's 0.1 threshold. Reserving DOM space with min-height containers dropped CLS to exactly 0 and moved iOS Safari checkout completion from 24% to 39%, recovering roughly $40,000 a month. Full breakdown in the Health and Wellness case study.

Neither of these problems would have surfaced from a heatmap. Both required someone to open the browser's Network tab and physically test on real hardware. That's the exact bar the questions below are designed to test for.

The 8 Questions to Ask Before Hiring

1. "When you audit a new store, what's the first tool you open?"

Red flag: "We start in Google Analytics" or "we run a heatmap first." Both are legitimate second steps. Neither is a first step if the goal is finding out why a store underperforms.

Green flag: Chrome DevTools, specifically the Network tab throttled to simulate a real connection, and a check of mobile versus desktop conversion rate before touching anything else. An agency that opens the browser before the dashboard is diagnosing the store, not just reading its symptoms.

2. "Do you audit for ghost scripts before recommending anything?"

Red flag: "What's a ghost script?" or a vague answer about "checking your apps." Uninstalling a Shopify app removes it from the dashboard. It does not remove the code the app wrote into your theme, and that leftover code keeps executing on every page load regardless of what the Apps page shows.

Green flag: They describe cross-referencing your installed apps against your actual network waterfall, looking for domains firing requests that don't match anything currently active. Dead scripts from deleted apps are one of the single most common, and most invisible, causes of slow load times we find in audits.

3. "Do you test on staging before pushing changes live?"

Red flag: "We usually push directly and monitor closely." That's a euphemism for testing on your live revenue.

Green flag: A documented staging or duplicate-theme process, with a specific QA checklist before anything touches production. This matters most for Shopify Plus stores where a single broken deploy during a high-traffic window can cost more than the entire engagement.

4. "Do you measure Core Web Vitals as a baseline, and what's an acceptable LCP?"

Red flag: "We focus on conversion rate, not technical metrics" or a number pulled with no context, like "under 4 seconds is fine."Google's own threshold for a good LCP is under 2.5 seconds, and a technically fluent agency should already know that number cold.

Green flag: A specific target under 2.5 seconds on mobile, stated without hesitation, tied to how load speed compounds into conversion loss.Akamai's research ties a 100ms delay in load time to a measurable drop in conversions, and an agency worth hiring should be able to cite that relationship without prompting.

5. "Do you know Liquid well enough to fix a rendering issue, or do you only work through apps?"

Red flag: "We'd recommend an app for that." If every proposed fix is a new app installation, the agency is treating your theme as a black box, and every new app adds more of the execution weight already dragging your store down.

Green flag: They can describe a real Liquid-level fix, in plain language, that removed friction without adding a dependency. Rebuilding a variant selector natively instead of relying on a third-party swatch app is the kind of answer that separates agencies who touch code from agencies who only touch settings pages.

6. "What did the previous agency or in-house team miss on your best case study?"

This is the failure-mode question from earlier in this post, asked directly. Red flag: a vague answer about "their strategy wasn't data-driven enough."

Green flag: a specific, named mechanism. In our own experience, this looks like: the previous vendor's proposal never mentioned Liquid, never tested on iOS Safari, or never opened DevTools before recommending a redesign. If the agency can name the exact gap in someone else's process, they can probably describe the gap in yours too.

7. "How would you catch a problem invisible on desktop but severe on mobile Safari?"

Red flag: "We test across browsers using an emulator." Emulators approximate rendering. They don't reproduce Safari's actual viewport and rendering timing, which is exactly why the checkout bug in the Health and Wellness case above was invisible to two previous vendors.

Green flag: Physical device testing, specifically a real iPhone in Safari, as a standard part of their QA process, not an occasional extra.

8. "If the technical audit runs long or turns up more than expected, what happens to the timeline and the invoice?"

Red flag: A flat monthly retainer with no connection between scope and what the audit actually finds. This is where agencies quietly bill for months of hypothesis-testing on a store that was never going to convert well until the underlying code was fixed.

Green flag: A clear answer about how findings shape the engagement, and comfort saying "we don't know exactly what we'll find until we look." That kind of honesty is rare, and it's a genuinely good sign.

The Practical Framework: Running This in a Real Call

You don't need all eight questions in every call. Ask questions 1, 2, and 6 first. If the answers are vague or app-focused, you already have your answer, and the remaining five questions won't change it.

If the first three answers are specific and technical, use questions 4, 5, and 7 to confirm depth rather than just vocabulary. Plenty of agencies have learned to say "Core Web Vitals" without being able to explain what's actually blocking your LCP. Ask them to walk through what they'd check first on your specific store, using your actual URL if you can share it. A genuinely technical agency will start naming things they'd expect to find, script bloat, an unoptimized recommendation carousel, a checkout widget loading late, before you've told them anything about your setup.

Save question 8 for after you've decided you like them. It's a contract-structure question, not a competence question, and it protects you either way.

If you want to run this diagnostic yourself before you even start interviewing agencies, our free technical audit gives you a baseline reading on your store's Core Web Vitals, script load, and checkout stability, so you walk into every one of these conversations already knowing what a competent answer should sound like. And before you sign anything, it's worth comparing the proposal against what a Shopify CRO engagement should actually include, so you know whether you're paying for a diagnosis or just a retainer.

The founder from the opening of this post didn't sign that twelve-page proposal. He asked the agency two of the questions above. Neither got a specific answer. He walked.

Article by

Muhammad Usama

Founder & Head Conversion Engineer

Founder & Head Conversion Engineer with 8+ years of technical engineering experience. I bridge the gap between full-stack development and e-commerce growth, specializing in tearing down Shopify architectures, eliminating code-level friction, and building high-performance infrastructure for 7- and 8-figure brands.