Our current take on product analytics

As someone who first entered the software engineering field as a frontend engineer, one of the first tickets that gets assigned to you is the “add tracking to feature X” or “figure out this analytics bug”. That work has probably been on the shelf for a little while, and with good reason.

I’ve been asked: why do we have 26 signups in our database, but our analytics only show 22? Or: you said you implemented tracking, but I can’t figure out how to get the number go up in this complicated Google Analytics dashboard!

This was the start of a relationship with analytics that made me question whether or not computers were deterministic at all.

I now understand why the 4 senior engineers in my team wisely kept looking at their screens while the Growth Hacker wandered into our area and started asking ‘can someone help me figure out why I can’t see any data in our analytics dashboards?’. Should’ve kept quiet, but before long I was on a video call with a tech from Google Analytics toggling a feature after diving in and out menus for 45 minutes.

Our thesis

I’ve worked as the founding engineer, in small scrum teams, and big orgs: the cost of analytics is not the cost of your analytics subscription: it’s the engineering time and cognitive overhead you put on yourself to frantically prove you’re doing the right thing.

You probably should use analytics, and then get on with building your product.

What tends to happen

Analytics is part of your product strategy, but it doesn’t get the attention it deserves and thus your tracking plan is an afterthought. It exists as a forgotten bullet point in your JIRA template’s definition of ready, or most probably as a spreadsheet somewhere outside your codebase.

At some point a PM, or analyst wakes up and as an overcorrection decides that, if there’s no plan, we’d better track everything. That way we don’t miss anything! That creates a lot of noise. Noise like: “Oh, you can close that modal by clicking outside it, the cross icon in the top right or the cancel button? We’d better track all of those actions individually and with their own distinct tracking event id!“. And god forbid we go live with this feature no-one is using and we miss the first day of tracking!

Kudos to Mixpanel for creating Autocapture which allows engineers to offload this bad strategy to the analysts themselves by tracking all clicks automatically and allowing you to label them afterwards. At least we free up some engineering time, but it does leave us with a poor tracking plan though. You postpone the pain to the moment you are trying to make sense of all of the clicks you’ve collected and how they lead up to some goal you have in mind.

What we want to know

The basics

We are building Permiso, an Identity Provider that solves Single Sign On (SSO) and other parts of Identity Access Management (IAM) for organizations. We have a public landing page, documentation and a private product for our customers. We need to know how many people visit our public sites and whether or not they interact with the most important thing there: do they use our demo application and do they leave their address behind for us to give them a call.

That core doesn’t need to change every week. What does change is the stuff we do around it.

Track broadly, not precisely

What did a blog post do to our traffic and, did it change the behaviour of people visiting our site? How does making edits to a well visited section of our documentation change their behaviour? Do they click through to try the product? Just like in weight loss: you should not be interested in the digits on the scale on any particular day, but your interest should be in the trend across a month.

Other than the very clear bottom line goals (signups, demo-uses) you probably know too little to formulate detailed hypotheses beforehand. Usage tracking should be broad enough so that it can be moulded into the questions that come to you later. By tracking page views and setting up goals beforehand you can figure out the broad strokes. After that, your analytics tool should help you query your data so you can filter out the noise and figure out what is happening in greater detail.

I’m sure that you can figure out what the best sequence of clicks is to track the entire journey of reading the homepage, clicking on read more and signing up at the end of a modal. Drawing that route on the map is very brittle as it will collapse the moment you make structural changes to your site. Which I expect to happen a lot right now.

Talk to your customers

At this point it’s so much more valuable to speak to a customer face to face. I’m not under the illusion that we only need to figure out the last 10% of our implementation and we have validated the other 90% is correct. We still need a lot of input from customers to figure out what the right components are for an identity provider. We believe it’s a simple capable core while extra modules are very personal. But what makes up that core and how often are these extra modules really needed?.

Maybe this is stating the obvious: we won’t figure out whether agentic role mining (whatever that might be) is the missing feature that is keeping your potentials from converting by tracking how many clicks the page explaining it has.

What we don’t need

Our biggest problem is that nobody knows who we are.

We should solve that and should not pretend we are a giant corporation trying to raise conversion with a few percentage points.

We are not a Nordic furniture store operating in a global market that can afford a month long experiment to test whether or not requiring to log in before a purchase increases sales in Germany. We don’t have the scale to single out that variable and come to statistical conclusions or compare bottom line results.

We don’t need to A/B test the copy of our button that says: “Try the demo” and we definitely don’t need to invest and maintain the instrumentation and code that’s required to get this up and running. Proving that you are capable of running A/B tests only proves you can run A/B tests.

We also don’t need or want mass surveillance to be able to track our customers across platforms to figure out how many of our ads they saw before clicking on our homepage.

There comes a time when all these things might be relevant to answer the (important!) concrete questions you have about operating your business. It’s not the first port of call when trying to figure out what your product should do.

Why Plausible

To sum up: we need a tool that’s easy to set up and easy to use. Where high-level stats are the default and zooming in on certain traffic and funnels is opt-in instead of having to codify your entire customer journey in an analytics tool.

We chose the hosted version of Plausible for our analytics. We feel good paying for software that we can audit, and where we are the customer instead of paying indirectly with the privacy of our visitors.

Plausible being built in the EU matters to us too seeing as we’re positioning Permiso as the local alternative, so it would be odd to route our own visitors through Google. We feel strongly about supporting good EU software alternatives.

In other projects we have used multiple versions of Google Analytics, Mixpanel and PostHog. They are either free (and that’s a bad thing because you are paying with your visitor’s privacy) or very expensive (kudos to PostHog for their generous free tier though) and all have way more settings, tools and add-ons than we need. If you’re looking for more concrete comparisons, Plausible’s own does a better job than I can here.

I’m personally thankful not having to build another cookie banner (which should have been a browser setting). And I sighed a genuine sigh of relief when I deployed the simple script snippet and got near instant feedback in their signup flow. We even got an email saying our first visitor just showed up (it might have been me though)!

There’s no Plausible-specific vocabulary: A website is just a website and a page view is a page view. I don’t need a separate tool to set up tracking (Google Tag Manager) and another to load in the analytics data (Google Analytics). And, as far as I’m aware we haven’t been assigned a dedicated Plausible rep that we need to call to debug our setup.

So we added some analytics, and got on with building the product.