What do product teams get wrong when trying to monetize open-source products that target developers?

One common mistake I have seen is having a monetization model that does not scale. Services and customization on top of open-source software will only scale as much as your workforce can. Having paid features on top of open source software and a tie ring philosophy that can be easily explained is esssential.

A tiering strategy for features based on buyers has worked well for GitLab. The free tier targets individual contributors, and other levels target enterprises. You can read more about it here https://about.gitlab.com/company/pricing/#three-tiers. This video where the GitLab CEO walks through a monetization strategy with a startup is also helpful https://www.youtube.com/watch?v=UbMkw6GD7TQ .

I have a lot of opinions about open-source business models, many of which I am sure are not popular, but let's jump in anyway. :-) First, let's start by acknowledging that there is a basic tension between open-source software (OSS), which is fundamentally socialist, versus our capitalist economic system. It is a very tricky balancing act for a PM, particularly if you have inherited a number of substantial, imperfect dynamics about the business from the founding team who have walked through certain one-way doors already. One of those one-way doors is the decision about what should be open-source or not, so the whole notion of trying to "monetize open-source products" is in no small way an attempt to reverse through a one-way door. This is why so much blowback occurs when companies attempt this (exhibit 1: Hashicorp changing to a Business Source License). Very, very few firms are successful without massive impact to their brand and destruction developer trust. I can only think of three in recent memory (MongoDB, Elastic, and Docker) who have pulled it off, and MongoDB already had the controversial AGPL moat to start off on a better footing. (Tenable is one of the few who were able to relicense Nessus decades ago and lived to see another day as a company, but the world has changed and such a move, made today, would cause an enormous storm relative to that which it caused back then.)

Assuming that the question is not about trying to force horses back into a barn after they've already escaped, the answer is that you have to figure out how to add commercial value on top of the open-source core, rather than trying to do a "take back" on the open-source by monetizing it just because you couldn't innovate on top of it. "Open core" business models are certainly controversial among the pure OSS types, but I would argue that they do work, except that you need to work twice as hard to create value on top of the OSS plus create a defensible moat. The best "open core" business models are ones that even the OSS zealots could live with: products that consume the data from OSS or make it better in some way (management interfaces, data analytics, etc.), but aren't directly exploiting the free labor of OSS contributors because there's a clear contract boundary between the OSS and the value-added product. The most problematic are ones where "core" is just a basic version of the enterprise product ("Community Edition" versus "Professional Edition") and the company is directly making money off all the free contributions from the OSS community.