From cd9c21e1cff6700d01753f330ebc8bfc4e706892 Mon Sep 17 00:00:00 2001 From: Luke Harries Date: Thu, 1 Dec 2022 14:46:06 +0000 Subject: [PATCH 1/7] added the vision --- .../2022-12-01-workflow-new-feature.md | 40 +++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 requests-for-comments/2022-12-01-workflow-new-feature.md diff --git a/requests-for-comments/2022-12-01-workflow-new-feature.md b/requests-for-comments/2022-12-01-workflow-new-feature.md new file mode 100644 index 0000000..783faab --- /dev/null +++ b/requests-for-comments/2022-12-01-workflow-new-feature.md @@ -0,0 +1,40 @@ +# The first "Workflow": Creating a new feature + +Owner: Luke Harries +Decision type: +Decision date: + +## What's a workflow? + +So far we have given product engineers the individual tools to create trends, create funnels, watch recordings, create feature flags etc. + +We have the opportunity to connect these flexible tools with opinionated best practices to focus on specific jobs-to-be-dones in PostHog. These could be called "Workflows". + +Two examples of potential workflows: + +1. Building a new feature - assisting with rollout and measuring success *More details below* +2. Workbooks - making it easy to come up with questions, save related insights, clip particular bits of recordings and jot down thoughts + +## Workflow: Creating a new feature + +Here's the vision of what this could be in 3 - 6 months: + +Sarah is a product engineer at a high-growth startup building a B2C website to help companies offset their carbon - called Carbon.io. She's working on a new feature for users to watch videos of their carbon-offsetting projects. She hypothesizes that users who watch these videos will feel more connected to the projects and retain at a higher rate. + +The first thing Sarah does is open up PostHog and start the workflow for "Create a new feature". It opens up a page to create the new feature where she names it "Videos of offsetting projects". Optionally, she can enter some of the background information on this feature including the hypothesis. She selects "auto-rollout" rather than "A/B test" - meaning once the feature is made it will be used internally and after manual approval will be rolled out to beta users then low-value users then high-value users. She selects the company KPI "user retention" as the key metric. + +Sarah clicks "Done". PostHog automatically creates a feature flag to control the rollout; sets up a dashboard containing session recordings of people using it; and creates the relevant graphs. + +Sarah builds the feature between the feature flag and merges it to master. + +Sarah pings her team to give it a go on a few different devices. When 5 internal users have used the feature PostHog sends her a slack message to the playlist. She reviews it and the errors logs - it's all working fine so she gives the thumbs up to escalate to beta users. There's no errors in beta and so it automatically deploys to prod - sending her a slack message to her team to keep them in the loop. PostHog automatically invites 2 users to a user interview after having used the video feature. + +From the dashboard, Sarah learns most of the users are only watching the first 30 seconds of the 5 minute videos. During the interviews she asks them what they thought of the videos and while they enjoyed them they said they were too long and dry. Sarah organizes for a new set of videos to be made and rolls out an update. + +3 months later Sarah gets a slack message from PostHog to check-in on the retention graph. They can see there's a significant correlation between retention between users that watched a video and users who retained. Success! + +### What are the benefits of building this? + +- Increase cross-product usage +- Become a core part of the engineering workflow +- Build in the best practices for PostHog and external engineers to use to build check for feature success without using PM time From 96e8c615983fb1b18ccd7dc5c0f461e2ab94fef6 Mon Sep 17 00:00:00 2001 From: Luke Harries Date: Thu, 1 Dec 2022 14:47:37 +0000 Subject: [PATCH 2/7] added decision context --- requests-for-comments/2022-12-01-workflow-new-feature.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/requests-for-comments/2022-12-01-workflow-new-feature.md b/requests-for-comments/2022-12-01-workflow-new-feature.md index 783faab..b28c535 100644 --- a/requests-for-comments/2022-12-01-workflow-new-feature.md +++ b/requests-for-comments/2022-12-01-workflow-new-feature.md @@ -1,8 +1,8 @@ # The first "Workflow": Creating a new feature -Owner: Luke Harries -Decision type: -Decision date: +Owner: Eric (team experimentation) +Decision to be made: Is this something we should build in Q1 as part of team experimentation? +Decision date: 7th December 2022 ## What's a workflow? From 10b9b22ed6a2114703ed8f49c6fbada13ddeae29 Mon Sep 17 00:00:00 2001 From: Luke Harries Date: Thu, 1 Dec 2022 14:56:52 +0000 Subject: [PATCH 3/7] added more info --- .../2022-12-01-workflow-new-feature.md | 34 +++++++++++-------- 1 file changed, 20 insertions(+), 14 deletions(-) diff --git a/requests-for-comments/2022-12-01-workflow-new-feature.md b/requests-for-comments/2022-12-01-workflow-new-feature.md index b28c535..3104cd1 100644 --- a/requests-for-comments/2022-12-01-workflow-new-feature.md +++ b/requests-for-comments/2022-12-01-workflow-new-feature.md @@ -1,9 +1,21 @@ -# The first "Workflow": Creating a new feature +# The first "Workflow": MVP for "Creating a new feature" -Owner: Eric (team experimentation) -Decision to be made: Is this something we should build in Q1 as part of team experimentation? +Owner: @EDsCODE (team experimentation) +Decision to be made: Should we build this in Q1 as part of team experimentation? Decision date: 7th December 2022 +## Context + +This RFC proposes that team experimentation builds the MVP of a workflow for "Creating a new feature" as part of their Q1 OKRs. + +It should sit with team experimentation as it would be a key driver of feature flag usage. + +This could have multiple benefits across the product: + +- Making PostHog the first place that product engineers go when building a new feature +- Increasing cross-product usage (feature flags, product analytics, session recordings) +- Encoding the best practices for PostHog and external engineers to use to build check for feature success without using PM time + ## What's a workflow? So far we have given product engineers the individual tools to create trends, create funnels, watch recordings, create feature flags etc. @@ -21,20 +33,14 @@ Here's the vision of what this could be in 3 - 6 months: Sarah is a product engineer at a high-growth startup building a B2C website to help companies offset their carbon - called Carbon.io. She's working on a new feature for users to watch videos of their carbon-offsetting projects. She hypothesizes that users who watch these videos will feel more connected to the projects and retain at a higher rate. -The first thing Sarah does is open up PostHog and start the workflow for "Create a new feature". It opens up a page to create the new feature where she names it "Videos of offsetting projects". Optionally, she can enter some of the background information on this feature including the hypothesis. She selects "auto-rollout" rather than "A/B test" - meaning once the feature is made it will be used internally and after manual approval will be rolled out to beta users then low-value users then high-value users. She selects the company KPI "user retention" as the key metric. +The first thing Sarah does is open up PostHog and start the workflow for "Create a new feature". It opens up a page to create the new feature where she names it "Videos of offsetting projects". Optionally, she can enter some of the background information on this feature including the hypothesis. She selects "auto-rollout" rather than "A/B test" - meaning once the feature is made it will be used internally and after manual approval will be rolled out to beta users then low-value users then high-value users. She selects "user retention" as the key metric. Sarah clicks "Done". PostHog automatically creates a feature flag to control the rollout; sets up a dashboard containing session recordings of people using it; and creates the relevant graphs. -Sarah builds the feature between the feature flag and merges it to master. - -Sarah pings her team to give it a go on a few different devices. When 5 internal users have used the feature PostHog sends her a slack message to the playlist. She reviews it and the errors logs - it's all working fine so she gives the thumbs up to escalate to beta users. There's no errors in beta and so it automatically deploys to prod - sending her a slack message to her team to keep them in the loop. PostHog automatically invites 2 users to a user interview after having used the video feature. - -From the dashboard, Sarah learns most of the users are only watching the first 30 seconds of the 5 minute videos. During the interviews she asks them what they thought of the videos and while they enjoyed them they said they were too long and dry. Sarah organizes for a new set of videos to be made and rolls out an update. +Sarah builds the feature between the feature flag and merges it to main branch. -3 months later Sarah gets a slack message from PostHog to check-in on the retention graph. They can see there's a significant correlation between retention between users that watched a video and users who retained. Success! +Sarah pings her team to give it a go on a few different devices. When 5 internal users have used the feature PostHog sends her a slack message to the playlist. She reviews it and the error logs - it's all working fine so she gives the thumbs up to escalate to beta users. There are no errors in beta and so it automatically deploys to prod - sending her a slack message to her team to keep them in the loop. PostHog automatically invites 2 users to a user interview after having used the video feature *Luke is working on an MVP of the user interviews feature - can be out of scope for the MVP*. -### What are the benefits of building this? +From the dashboard, Sarah learns most of the users are only watching the first 30 seconds of the 5-minute videos. During the interviews, she asks them what they thought of the videos and while they enjoyed them they said they were too long and dry. Sarah organizes a new set of videos to be made and rolls out an update. -- Increase cross-product usage -- Become a core part of the engineering workflow -- Build in the best practices for PostHog and external engineers to use to build check for feature success without using PM time +3 months later Sarah gets a slack message from PostHog to view on the retention graph. They can see there's a significant correlation between retention between users that watched a video and users who retained. Success! From df65894e52db0b325c26601cd5a6fd3812311f8b Mon Sep 17 00:00:00 2001 From: Luke Harries Date: Thu, 1 Dec 2022 15:03:13 +0000 Subject: [PATCH 4/7] added annikas feature description list --- .../2022-12-01-workflow-new-feature.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/requests-for-comments/2022-12-01-workflow-new-feature.md b/requests-for-comments/2022-12-01-workflow-new-feature.md index 3104cd1..89aea4f 100644 --- a/requests-for-comments/2022-12-01-workflow-new-feature.md +++ b/requests-for-comments/2022-12-01-workflow-new-feature.md @@ -27,7 +27,7 @@ Two examples of potential workflows: 1. Building a new feature - assisting with rollout and measuring success *More details below* 2. Workbooks - making it easy to come up with questions, save related insights, clip particular bits of recordings and jot down thoughts -## Workflow: Creating a new feature +## Vision for "Creating a new feature" workflow Here's the vision of what this could be in 3 - 6 months: @@ -44,3 +44,12 @@ Sarah pings her team to give it a go on a few different devices. When 5 internal From the dashboard, Sarah learns most of the users are only watching the first 30 seconds of the 5-minute videos. During the interviews, she asks them what they thought of the videos and while they enjoyed them they said they were too long and dry. Sarah organizes a new set of videos to be made and rolls out an update. 3 months later Sarah gets a slack message from PostHog to view on the retention graph. They can see there's a significant correlation between retention between users that watched a video and users who retained. Success! + +## Feature description + +1. PostHog should be the starting point when you create a new feature. As the first step, you will create a feature flag. Potentially a VSCode integration can help with this. +2. When the feature is ready for Beta testing, the engineer can enable the feature for a beta tester cohort or select a pre-defined rollout schedule. This way, he doesn’t have to decide who to release it to on a case-by-case basis. +3. Users can subscribe to feature flags, so when changes have been made to a flag, e.g. a rollout, they will be notified. This way, everyone else in the team knows about the status of a feature release. +4. When a feature is rolled out, PostHog automatically monitors relevant metrics and regressions. If those are above a certain threshold, the rollout is stopped and the relevant people are informed. +5. On a flag itself, there is information on how many users are currently on one of the variants and when this information was last fetched. The idea here is that we distinguish whether a flag is rolled out according to the PostHog interface, but it is in fact not available in the code (anymore). +6. Once a feature flag has been rolled out to the whole user base for x days/weeks, the relevant people are informed to archive the feature flag to prevent tech debt. From cd58c787f83441dd5953bfd1c0b6caff6feffa49 Mon Sep 17 00:00:00 2001 From: Luke Harries Date: Thu, 1 Dec 2022 15:06:53 +0000 Subject: [PATCH 5/7] added annikas feature description --- .../2022-12-01-workflow-new-feature.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/requests-for-comments/2022-12-01-workflow-new-feature.md b/requests-for-comments/2022-12-01-workflow-new-feature.md index 89aea4f..c10ef52 100644 --- a/requests-for-comments/2022-12-01-workflow-new-feature.md +++ b/requests-for-comments/2022-12-01-workflow-new-feature.md @@ -16,6 +16,8 @@ This could have multiple benefits across the product: - Increasing cross-product usage (feature flags, product analytics, session recordings) - Encoding the best practices for PostHog and external engineers to use to build check for feature success without using PM time +*This is context for the engineers leading it and should be iterated* + ## What's a workflow? So far we have given product engineers the individual tools to create trends, create funnels, watch recordings, create feature flags etc. @@ -39,15 +41,15 @@ Sarah clicks "Done". PostHog automatically creates a feature flag to control the Sarah builds the feature between the feature flag and merges it to main branch. -Sarah pings her team to give it a go on a few different devices. When 5 internal users have used the feature PostHog sends her a slack message to the playlist. She reviews it and the error logs - it's all working fine so she gives the thumbs up to escalate to beta users. There are no errors in beta and so it automatically deploys to prod - sending her a slack message to her team to keep them in the loop. PostHog automatically invites 2 users to a user interview after having used the video feature *Luke is working on an MVP of the user interviews feature - can be out of scope for the MVP*. +Sarah pings her team to give it a go on a few different devices. When 5 internal users have used the feature PostHog sends her a slack message to the playlist. She reviews it and the error logs - it's all working fine so she gives the thumbs up to escalate to beta users. There are no errors in beta and so it automatically deploys to prod - sending her a slack message to her team to keep them in the loop. PostHog automatically invites 2 users to a user interview after having used the video feature *Luke is working on an MVP for the user interviews feature - can be out of scope for the MVP*. From the dashboard, Sarah learns most of the users are only watching the first 30 seconds of the 5-minute videos. During the interviews, she asks them what they thought of the videos and while they enjoyed them they said they were too long and dry. Sarah organizes a new set of videos to be made and rolls out an update. -3 months later Sarah gets a slack message from PostHog to view on the retention graph. They can see there's a significant correlation between retention between users that watched a video and users who retained. Success! +3 months later Sarah gets a slack message from PostHog to view the retention graph. They can see there's a significant correlation between retention between users that watched a video and users who retained. Success! -## Feature description +## Potential features -1. PostHog should be the starting point when you create a new feature. As the first step, you will create a feature flag. Potentially a VSCode integration can help with this. +1. PostHog should be the starting point when you create a new feature. As the first step, you will create a feature flag. Potentially a VSCode integration or CLI tool can help with this. 2. When the feature is ready for Beta testing, the engineer can enable the feature for a beta tester cohort or select a pre-defined rollout schedule. This way, he doesn’t have to decide who to release it to on a case-by-case basis. 3. Users can subscribe to feature flags, so when changes have been made to a flag, e.g. a rollout, they will be notified. This way, everyone else in the team knows about the status of a feature release. 4. When a feature is rolled out, PostHog automatically monitors relevant metrics and regressions. If those are above a certain threshold, the rollout is stopped and the relevant people are informed. From de82ff27499eff45996ffac040f6acc3e4321c08 Mon Sep 17 00:00:00 2001 From: Luke Harries Date: Fri, 2 Dec 2022 14:28:02 +0000 Subject: [PATCH 6/7] Update requests-for-comments/2022-12-01-workflow-new-feature.md Co-authored-by: Annika <14750837+annikaschmid@users.noreply.github.com> --- requests-for-comments/2022-12-01-workflow-new-feature.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requests-for-comments/2022-12-01-workflow-new-feature.md b/requests-for-comments/2022-12-01-workflow-new-feature.md index c10ef52..efc8826 100644 --- a/requests-for-comments/2022-12-01-workflow-new-feature.md +++ b/requests-for-comments/2022-12-01-workflow-new-feature.md @@ -41,7 +41,7 @@ Sarah clicks "Done". PostHog automatically creates a feature flag to control the Sarah builds the feature between the feature flag and merges it to main branch. -Sarah pings her team to give it a go on a few different devices. When 5 internal users have used the feature PostHog sends her a slack message to the playlist. She reviews it and the error logs - it's all working fine so she gives the thumbs up to escalate to beta users. There are no errors in beta and so it automatically deploys to prod - sending her a slack message to her team to keep them in the loop. PostHog automatically invites 2 users to a user interview after having used the video feature *Luke is working on an MVP for the user interviews feature - can be out of scope for the MVP*. +Sarah pings her team to give it a go on a few different devices. When 5 internal users have used the feature PostHog sends her a slack message to the playlist. She reviews it and the error logs - it's all working fine so she gives the thumbs up to escalate to beta users. There are no errors in beta and so it automatically deploys to prod as a phased release - sending her a slack message to her team to keep them in the loop. PostHog automatically invites 2 users to a user interview after having used the video feature *Luke is working on an MVP for the user interviews feature - can be out of scope for the MVP*. From the dashboard, Sarah learns most of the users are only watching the first 30 seconds of the 5-minute videos. During the interviews, she asks them what they thought of the videos and while they enjoyed them they said they were too long and dry. Sarah organizes a new set of videos to be made and rolls out an update. From 0a466d006204ae23e304d2cad740e46f2437de34 Mon Sep 17 00:00:00 2001 From: Luke Harries Date: Thu, 8 Dec 2022 11:33:17 +0000 Subject: [PATCH 7/7] Added more workflows --- requests-for-comments/2022-12-01-workflow-new-feature.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/requests-for-comments/2022-12-01-workflow-new-feature.md b/requests-for-comments/2022-12-01-workflow-new-feature.md index efc8826..57fe692 100644 --- a/requests-for-comments/2022-12-01-workflow-new-feature.md +++ b/requests-for-comments/2022-12-01-workflow-new-feature.md @@ -24,10 +24,13 @@ So far we have given product engineers the individual tools to create trends, cr We have the opportunity to connect these flexible tools with opinionated best practices to focus on specific jobs-to-be-dones in PostHog. These could be called "Workflows". -Two examples of potential workflows: +Some examples of potential workflows: -1. Building a new feature - assisting with rollout and measuring success *More details below* +1. Creating and shipping a new feature - assisting with rollout and measuring success *More details below* 2. Workbooks - making it easy to come up with questions, save related insights, clip particular bits of recordings and jot down thoughts +3. Iterating on existing features - making sure the new feature is better than the previous iteration +4. Conversion experiment - making sure that the funnel metrics improve (including quality of new customers) +5. Fixing bugs - understanding what's broken and confirming that the fix works ## Vision for "Creating a new feature" workflow