Ask HN: How to you track if newly released features have expected engagement?

2 pointsmakwarth3y ago10 comments

When launching new features, how do you track them and how do you measure their "success"?

Do you use a tracking plan, like Segment Protocols? How do you determine when the feature is done (doesn't need more iteration)?

Disclaimer: I'm the founder of a startup that is focused on feature analytics. It's still early days for us, so here to learn about the existing pains in this space.

10 comments

davikr3y ago

I can't offer any advice, but as an user, I urge for features not to be removed due to a lack of engagement, as was done for Firefox's "compact mode" for some time. I think they might have reversed that change.

makwarthOP3y ago

Good point! Talking to some of the users of the feature you're about to remove is probably a good first step. Do those users use the feature in a way that's aligned with your product strategy?

jesuscript3y ago

Users will always find edge case bugs and use cases the feature does not fully handle. Feedback will come for free if there is engagement.

If you hear nothing for a new feature, then no one touches it.

makwarthOP3y ago

True ;) Sometimes, though, a new feature isn't getting engagement because people are busy. Product and Product marketing can use low engagement metrics to raise awareness.

Raed6673y ago

A/B test: Do users who have the feature enabled get you better KPIs or not.

makwarthOP3y ago

How do you track that exactly? Any chance you can share a concrete example?

Raed6673y ago

Randomly, take 10%-25% of your users, show them the feature and see if they do more of what you want them to do.

If you're blogging platform, that would be writing more blogs, or whatever metric that guides your business growth.

If yes, then roll-it out to 50%, confirm that the metrics are moving to the same direction and by how much. You should be able to answer the question: "users who have this feature do $X more/less/faster/etc.."

You can implement this all in house or use one of the tens of A/B testing SaaS out-there.

1 more reply

j / k navigate · click thread line to collapse

Ask HN: How to you track if newly released features have expected engagement?

2 pointsmakwarth3y ago10 comments

When launching new features, how do you track them and how do you measure their "success"?

Do you use a tracking plan, like Segment Protocols? How do you determine when the feature is done (doesn't need more iteration)?

Disclaimer: I'm the founder of a startup that is focused on feature analytics. It's still early days for us, so here to learn about the existing pains in this space.

10 comments

davikr3y ago

makwarthOP3y ago

Good point! Talking to some of the users of the feature you're about to remove is probably a good first step. Do those users use the feature in a way that's aligned with your product strategy?

jesuscript3y ago

Users will always find edge case bugs and use cases the feature does not fully handle. Feedback will come for free if there is engagement.

If you hear nothing for a new feature, then no one touches it.

makwarthOP3y ago

True ;) Sometimes, though, a new feature isn't getting engagement because people are busy. Product and Product marketing can use low engagement metrics to raise awareness.

Raed6673y ago

A/B test: Do users who have the feature enabled get you better KPIs or not.

makwarthOP3y ago

How do you track that exactly? Any chance you can share a concrete example?

Raed6673y ago

Randomly, take 10%-25% of your users, show them the feature and see if they do more of what you want them to do.

If you're blogging platform, that would be writing more blogs, or whatever metric that guides your business growth.

You can implement this all in house or use one of the tens of A/B testing SaaS out-there.

1 more reply

j / k navigate · click thread line to collapse