Need Feedback on Your App? Do This First
Here’s how we pulled off a high-impact closed beta (and how you can too) with a scrappy team, zero budget, 6 weeks timeline and 230 testers
Hi 👋! You are reading one of my free articles - I hope it helps you level up your PM skills. Want to go deeper? My premium subscribers get exclusive frameworks and strategies I don’t share anywhere else - content that has helped hundreds of product managers land better roles and make bigger impact.
Ready to accelerate your product career? Subscribe now for premium access and join a community of exceptional PMs gaining an edge every week.
Now, let’s dive into today’s insights…
Let’s face it, most beta testing advice assumes you have an unlimited budget, a lot of time, or even both. But what if you have neither? Then this piece is for you.
When we started building Project Quartz, our goal was deceptively simple. It was to fix calendar chaos for remote teams. The kind where meetings overlap, no one knows who’s available, and scheduling across time zones feels like defusing a bomb.
We had three engineers, one PM, a loose prototype across macOS, iOS, Android, and 45 days to validate the product before committing to web and Windows.
This is how we recruited 230+ testers across 20 industries and turned raw feedback into roadmap clarity with zero marketing spend.
Folks, if you are building something people might need, but you need to prove they need it, this is your playbook.
Why We Ran a Closed Beta
The purpose wasn’t to polish or build the next shiny thing; it was mainly proof. And before investing further in Quartz, we needed to answer four key questions:
Is the problem real and painful?
Do our early flows resonate, or confuse?
Where does Quartz fit into current workflows?
What’s missing for this to become a daily user habit?
This wasn’t about collecting feature requests. It was about pattern recognition. What themes emerge when 200+ people use your product in the wild?
The Stack: Our Free Toolchain
We didn’t wait for tooling sign-off. We used what we could spin up fast:
Landing Page: Tilda (clean, no-code, live in hours).
Surveys: SurveyPlanet (good branching logic, minimal friction).
Scheduling: Calendly (free trial, but mostly skipped in favor of async).
Calls: Skype (old-school but reliable).
Email Outreach: Reply.io (free trial, strong deliverability).
Support: Zendesk (testers loved in-app ticketing).
Key mindset: Our goal wasn’t tooling perfection. It was getting a signal with velocity.
So, how did we get 230 testers without spending a dime? Here’s the breakdown.
We used three outreach channels:
Past users of our previous products.
Beta listing platforms like Betabound.
Targeted posts on Reddit, Product Hunt, and Discord Groups.
Rather than posting to generic beta tester subs, we went where our potential users already hung out — indie startup founders, operations managers, freelancers juggling multiple calendars. Subreddits like r/RemoteWork and r/Productivity, plus niche newsletters and Slack groups.
We didn’t lead with “Help us test our product.”
We led with, “Are you struggling with back-to-back calls? We might have something for you.”
Conversion rate from signup to full participation? Over 60%.
The Structure: 2 Loops, 3 Weeks
The beta lasted 3 weeks and ran as two tight feedback loops.
Week 1–2: Initial Exposure
Users completed a screener (role, calendar pain points, preferred platforms).
We gave them a free 1-year license to Quartz.
They used the product organically for ~10 days.
Week 2–3: Feedback + Deeper Usage
Users chose an async survey or an optional 15-minute call.
We followed up with summaries, recordings (when allowed), and clarified any confusing inputs.
We adjusted survey logic mid-flight based on early responses.
What We Learned
You don’t need 1,000 responses to get clarity. We saw strong, repeating patterns that shaped our next sprint.
The top insights were:
Over 40% of users still used screenshots or manual time zone math to coordinate meetings.
Quartz’s auto-timezone visualizer was an immediate differentiator.
Slack and email were the top places where meetings broke down.
This gave us confidence to invest in integrations and shared availability links.
People loved the minimal UI but hated the onboarding friction.
We introduced a 90-second guided setup in the next build. Adoption jumped 22%.
Two personas emerged fast:
The Coordinator: Ops, admin, or EA roles, organized everyone else’s chaos.
The Switcher: Freelancers, consultants, juggling clients across time zones, tools, and platforms.
Tactical Wins
Surveys outperformed calls (80:20 ratio), but calls gave richer insights.
Follow-ups mattered: When we responded to survey feedback with thoughtful questions, users engaged deeper.
Mandatory questions weren’t a blocker if the value was clear.
We reused feedback in marketing copy, pitch decks, and even onboarding tooltips.
If you are wondering what we did after the Beta, we didn’t disappear. Instead, we:
Sent a thank-you video from the team.
Included behind-the-scenes screenshots from user suggestions we’d implemented.
Invited power users to gain early access for the open beta.
Collected testimonials (many wrote them unsolicited after the follow-up).
Most importantly, we knew what to build next and what not to.
Key Takeaways for Scrappy Product Teams
Start narrow. Don’t try to validate every feature or user type.
Treat testers like co-creators. It builds trust and stronger feedback.
Live calls are gold, but only after you've earned trust.
Async feedback scales. But structure your surveys well.
End with gratitude. A thank-you goes a long way.
TL;DR; Closed Beta Blueprint
Define what you’re validating.
Use free AI tools to get moving fast.
Go where your users already are, not where beta testers hang out.
Run two usage loops: test, feedback, refine.
Follow up. Close the loop. Show you listened.
Use the insights to shape the roadmap, positioning, and onboarding.
Listen, you don’t need a big brand or budget to run a high-impact closed beta. You need clarity, care, and execution. Heck, there are AI prototyping tools that will accelerate putting visuals in front of your users faster than ever.
If you’re building something new, build like this.
ICYMI: How do you launch a 0–1 product to a group of cult users—and turn it into something they use every day? If you’re a PM bringing a new product to market for the first time, this is just one of many questions you should be asking. Fortunately, these articles dive into this exact moat-building challenge 👇🏾
Launching a product goes beyond just
pushingfrom sandbox to app store. There are multiple steps involved and a host of cross-functional stakeholders you’ll work with that’s outside the product team. This piece outlines five essential steps for launching a successful product—from dev team to legal.
The difference between an app people use daily and one they abandon after a single use is stickiness. One becomes part of their routine; the other fades into the background. So how do you build stickiness into your product right from development—before users even get their hands on it? I share the secret in this piece.
