Introduction

There is a big focus on simplifying the product for Buffer users on free plans in 2017 in order to create a more intuitive experience and encourage more upgrades.

The purpose of this analysis is to get a better understanding of how users currently interact with the free plan, in order to better inform decisions around future plan limits. Specifically, we would love to learn about how many profiles users connect and actually schedule updates for.

More data questions and context can be found in this Paper, and potential changes to the Free plan can be found in this Paper.

Methodology

To answer this question, we’d like to only consider users that are currently active, and analyze their usage during the period in which they were active in Buffer. In this case, active is defined as having scheduled at least one update with Buffer in the previous 28 days.

We’ll gather all of the currently active users that are on free plans. Then, we will gather data on the number of profiles they have connected and the number of updates that they have scheduled for each profile.

Once we have this data, we will gather some summary statistics on the number of active profiles for each active user, and visualize the distribution of the number of profiles in use. It is unlikely to be normally distributed (bell shaped), so visualizing this breakdown would be useful to understanding the potential impact that changing the limits might have.

Data collection

Let’s start by getting the Free users that have scheduled at least one update in the past 28 days. We’ll use the following SQL query to grab them.

with user_facts as (
  select
    up.user_id
    , max(up.created_at) as last_update_created_at
    , count(distinct p.profile_id) as number_of_profiles
  from transformed_updates as up
  inner join users
    on users.user_id = up.user_id
  left join profiles as p
    on p.user_id = users.user_id
  where up.status != 'service'
  and users.billing_plan = 'individual'
  and p.is_disabled = false
  and p.is_deleted = false
  group by up.user_id
)
select
  user_id
  , last_update_created_at
  , number_of_profiles
from user_facts
where last_update_created_at > (current_date - 29)

There are around 250 thousand users in this dataset! Now we need to get the number of updates they have scheduled, and the number of profiles that they have scheduled updates for in the past 28 days.

select
  up.user_id
  , count(distinct up.id) as updates_count
  , count(distinct up.profile_id) as profile_count
from transformed_updates as up
left join users
  on up.user_id = users.user_id
where users.billing_plan = 'individual'
and up.status != 'service'
and up.created_at > (current_date - 29)
group by up.user_id

Awesome! Now let’s join the updates and users dataframes so that we’ll have the number of updates and active profiles for each user. We’ll use dplyr’s left_join function and replace NA values with 0.

# Join updates and users
users <- users %>%
  left_join(updates, by = 'user_id')

# Replace NAs with 0s
users[is.na(users)] <- 0

Exploratory analysis

Now that we have the updates and profile counts for each active user, we can compute some quick summary statistics for our population. Let’s first only look at the number of profiles as these currently active free users have connected.

# Summarise the number of profiles
summary(users$number_of_profiles)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   1.000   1.000   2.000   2.175   3.000  95.000

This summary and bar chart tell us a few important factoids.

• Around 38% of currently active free users have 1 (non-disabled) profiles connected.
• Around 25% of currently active free users have 2 (non-disabled) profiles connected.
• Around 21% of currently active free users have 3 (non-disabled) profiles connected.
• Around 12% of currently active free users have 4 (non-disabled) profiles connected.
• Around 37% of currently active free users have more than 2 (non-disabled) profiles connected.
• Around 16% of currently active free users have more than 3 (non-disabled) profiles connected.

The 37% of users that have more than 2 profiles connected equates to around 92 thousand users. That is a large number of active users that would be affected if we changed the profile limit to 2 profiles.

If instead we changed the profile limit to 3, around 16% of active free users would be affected, which equates to around 40 thousand users.

In this case we’re only considering profiles that active users have connected to Buffer. What if instead we only looked at the number of profiles that these users have scheduled updates for in the past 28 days? We can do this by looking at the profile_count dimension, which was derived from the updates table.

# Get summary stats on the number of profiles
summary(users$profile_count)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   0.000   1.000   1.000   1.688   2.000 150.000

Here is the breakdown:

• Around 50% of active Free plan users scheduled updates for only 1 profile in the past 28 days.
• The average number of profiles that active Free plan users used in the past 28 days is around 1.9.
• Around 75% of active Free plan users scheduled updates for only 2 or less profiles.
• Someone scheduled updates for 194 profiles in the past 28 days!

The maximum of 194 brings up a good point about our sample: these are only users that are currently on individual plans. This means that they could have been on a paid plan in the past 28 days. I don’t think it’s a huge portion, or very significant for this particular analysis, but it is something to keep in mind.

Let’s look at the distribution of profiles for active free users.

Wow, that’s interesting! It looks like most currently active Free users have only scheduled updates to one profile in the past 28 days.

One possibility is that there are very new users that have only gotten the chance to schedule updates to one or two profiles. Let’s see what would happen if we only looked at users that have sent at least 5 updates in the past 28 days.

We can see here that the proportion of active users that only schedule to a single profile changes quite a bit, but it is still high at 46.5%.

Let’s look at the CDF of the profiles distribution.

Interesting! This graph tells us the following:

• Around 56% of users scheduled updates to 1 profile in the past 28 days.
• Around 76% of users scheduled updates to 2 or less profiles in the past 28 days.
• Around 91% of users scheduled updates to 3 or less profiles in the past 28 days.
• Around 98% of users scheduled updates to 4 or less profiles in the past 28 days.

Conclusions

There are a couple important considerations to be made based on this data. Over 90 thousand users, or around 37% of the active free user population, has more than 2 profiles connected to Buffer as of June 2017. Around 40 thousand users, 16% of the active free user population, has more than 3 profiles connected. These are significant portions of Buffer’s userbase.

We could consider that users don’t necessarily use all of the profiles that they have connected to Buffer. For example, a user could have 4 profiles connected and only schedule updates for a single twitter account. The data suggests that this happens fairly often – around 76% of active free users only scheduled updates to two or less profiles in the past 28 days.

However, the fact that many users haven’t scheduled updates for all of their profiles recently in Buffer doesn’t necessarily mean that they would react or be affected by these changes to the profile limits.

We can assume that a certain amount will upgrade, but a certain amount will also leave Buffer. We also don’t quite know what effect word-of-mouth will have. We also may make the assumption that people with only 1 or 2 profiles will be unaffected – this may not necessarily be the case. We don’t fully understand the underlying causal effects that the profile limits have.

One way to get a better understanding of those causal mechanisms would be to run an experiment. If we could adjust the profile limits of a subset of our active Free user population, and allow them to upgrade to increase their limits, we would be able to make a better estimate of how this change might affect our user base and revenue stream. I would suggest running an experiment on a small subset of our active Free user population before making any definitive statement on the effect it will have on revenue.

Given the large number of people that would be affected, I would strongly suggest running experiments to better understand how free users and new users would react to these changes in profile limits.