(Organic) Groups in Four Voices

This post is going to be somewhat unusual, as it’s being written by four people. It’s going to be on a single topic (kind of) of having groups functionality, under a single code base and database.

It’s notably about Organic groups for Drupal 8, but not just. In fact, I think that at this point in time, my story about it, is just one out of three. Pieter Frenssen and Maarten Segers, the co-maintainers have their own voice.

But to make this one even more complete (and interesting), we should hear also Kristiaan Van den Eynde’s voice – the author of Group – a competing/complementing module.

Coincidentally and apart from me, all those people heavily invested in group functionality are Flemish. So, if you ever wondered how the story of “Three Flemish and an Israeli walk into a Bar,” goes… Here’s your chance.

Amitai Burstein

Ten years ago, I did my very first DrupalCon presentation about how the UN approached Gizra, my company, to ask us to build them a Drupal site, powered by OG to serve all UN country members.

The presentation video is of poor quality, which is in line with my poor acting and low talent for imitations. I watched it just recently and while I did occasionally laugh at my own past-me jokes, I was cringing at the rest. I sure had a lot of nerve at that time. I remember that I came to it with the “Make it or break it” attitude.

I think it succeeded, in the sense that overnight, I – along with Gizra – became more known in the community. It surely helped landing a few big contracts. But still, in one of those cases of “I didn’t see this was coming” reality is so good at, in 2019 Gizra is really building that UN site, and it’s really powered by OG8. So yeah, now on “Truth or Lie” games I can say “I’ve built a site for Kim Jong-un” and get away with it.

Here are some of my thoughts about Drupal, OG8 and being a maintainer of a popular module:

1. My concept of group functionality has not changed much in the past ten years. Framing the problem is quite simple: have permissions and membership on the group level, similar to how one has it on the site as a whole. The implementation, or rather the flexible implementation is where it gets complex.

I’d argue that one can build a very hard-coded group functionality in a short time, in a similar way I’m constantly tempted to think that I could build an e-commerce site with hard-coded and limited functionality in a short time. Maybe I could. But nowadays I know it often ends badly. OG’s value (or to that matter Group’s, as-well) isn’t just the code itself. That could change from one version to another. The biggest value is what I like to call “codified knowledge base.”

Almost everything I and the rest of the co-maintainers know, is somewhere in the code. It’s not just about having the right IF and ELSE. It’s making sure all those pieces of knowledge and lessons learned, are dumped from our brains into a more Git-friendly version.

2. Drupal – the CMS is quite amazing. It’s surely one of the best CMS out there. But like every big project it has many moving parts.

I believe that my biggest concern has always been trying to avoid breaking stuff once they are released. Automatic testing is a big part of it, and having a quite big test coverage is what allows us to code and refactor. Over the years, I have grown more and more fond of and familiar with static-typed languages, and friendly compilers. As a result, I’ve become more and more afraid of doing those refactors in PHP.

Since I’m not a purist, and rewriting Drupal to use [enter-your-favorite-statically-typed-language] is not an option, I’ve also learned to overcome these fears. Write code; Write tests; Cross fingers; Rinse and Repeat.

3. Drupal – the community is quite amazing. It’s surely one of the best communities out there. No buts.

While I have been less involved with the actual coding of OG8 in the past couple of years, I keep reading almost every single PR. This is such a great learning experience for me. From going over Pieter’s well-thought, well-implemented, and crazily-tested; to learning Maarten’s techniques for code review – manual and automatic ones.

It’s also interesting to see what I’ve felt for some time now, being put into numbers in Dries’s post Who sponsors Drupal development? :

Most contributions are sponsored, but volunteer contributions remains very important to Drupal’s success.

OG8, unlike the days of OG7, isn’t being built by someone(s) spending their evenings on writing features and fixing bugs of community “strangers”. It’s done by senior developers working for organizations that sponsor their work (and of course, some of their spare time, which I value even more!)

This means that currently OG8 is a snapshot of different needs, and almost nothing more. As crazy as it may sound, the UI for setting permissions for OG roles isn’t yet there. It’s surely needed by others, but not enough for a big organization to sponsor it. I suspect this means that while in the past OG was adopted by site builders, nowadays it more oriented towards developers. I’d want that eventually it will allow site builders to set it up – but evidently I don’t want it enough to do it in my spare time.

4. I’ve been with OG for a long time – since Drupal 4.7. First as a site builder, then slowly started to learn how to code, and submitted patches, and then I became the maintainer, and then I have let others join the ride.

I love the fact that I was able to evolve inside this little island, and it’s still an interesting journey. Before Drupal 8 was released, for a year or two I have felt a bit bad for not having it more in me to giving OG8 more love. I was noticing how Group module was gaining momentum, while OG wasn’t advancing as fast as I was used to. It was probably one of the very few times I had that contribution (and lack of it) to open source caused me some inconvenience. I wasn’t burned out or too stressed, just thinking about OG didn’t give me as much pleasure as it once did.

But along those two years I have also done some more significant changes in my life/work style. Starting to work at home (which I love); Working less hours (which I adore); And training 5-6 times a week (which I can’t stop talking about).

I think that every maintainer of a popular module reaches at a certain point this cross road, asking themselves how to continue. Finding that spot I currently have, which is more of – let’s make sure the ship is heading the right way, and not falling into pitfalls I know for so long – is I believe valuable for the project, but also for me as a person.

Pieter Frenssen

If I would make a list of all the skills I have learned in a decade of working with Drupal then “contributing to open source” would be right at the top. I have been an enthusiastic user of open source software for many years, but I thought it would be impossible for a self-taught programmer like me to make any meaningful contributions to these software packages I was using every day. Among a bunch of other software I used Drupal to build my websites, but – apart from reporting the occasional bug in the issue queue – I did not make any contributions. But this changed at my first Drupal conference.

As many people in our field I was originally working in a different industry. I am an audio engineer by trade, and I spent the first 15 years of my career in recording studios and on concert stages. Programming was a fun hobby and a great way to pass the long waiting periods in between concerts. Over time, I got more and more demand for creating websites and at some point I realized it was time to take my hobby more seriously. I needed to get in touch with other programmers and a conference seemed like a great way to do this.

Drupalcon Copenhagen was the first time I attended a tech conference and it was an eye opening experience. Here I met the people whose names I saw in the issue queue, and not only were they giving presentations to share their knowledge, but also actively working on building the software. And they were very happy to include newcomers. I was suddenly taking part in discussions and was invited to work on fixing some bugs in the upcoming Drupal 7. This was very interesting and tons of fun.

Back home I started to regularly contribute small fixes for bugs that I stumbled upon. It was in this period that I made my first contributions to Organic Groups: I fixed some small bugs like PHP notices and mistakes in the help texts. But apart from these my involvement with OG in the Drupal 7 era was as an end user. I made several projects with it, and interestingly enough one of those projects was done in collaboration with Kristiaan Van den Eynde who later went on to create the Group module based on his vision of how groups were supposed to work.

When Drupal 8 came around, my team was evaluating the modules we needed to have for porting a large Drupal 6 based project to Drupal 8. There were a few modules that already had a stable Drupal 8 version, but most were either in alpha or beta, or did not even have a release yet. One of those modules was Organic Groups. At the time it was not more than a proof of concept and most of the functionality that was used in our D6 project was not there yet. We decided to go ahead with it, and implement any functionality that we needed from OG in scope of our project. There were a few factors that made this not as crazy as it might seem:

There was a large time frame involved. The project is large and very complex, having been in continuous development for 8 years. It was estimated that the D8 port – including a completely new backend architecture – would take around 18 months of time. Any work that we would put towards improving Organic Groups on Drupal 8 would benefit our organisation in the future. We have a large number of websites using Organic Groups on Drupal 7 which will need to be migrated to D8 at some point in the future. If we were only considering a single site it would have been more economical to migrate to Group instead.

Organic Groups is one of the most complex modules in the Drupal ecosystem, and has evolved over a period of 15 years, being first conceived in 2004. It implements many edge cases that came out of real life use of the module in tens of thousands of different websites. And now it needed to be ported to Drupal 8. How do you approach a task like this? Our project manager did not like the idea of having a Jira ticket called “Port OG to D8”, estimated to 500 hours, and locking away a developer for a couple of months. So we decided on the following:

• We would contribute work in small, well scoped PRs. Small PRs are easier to review and accept by Amitai and his team of co-maintainers. We considered that this would reduce the risk that our sprint goals would be affected by a PR not being accepted in time. There was always a direct link between a PR in the Organic Groups issue queue and a ticket on our Jira board regarding some business objective.

• Try to be as clear as possible in the description of the PR about what the change is intended to achieve. If there was any doubt about the approach to take, we would create a discussion issue before starting any development. On our side, the Jira ticket describing the business need could then be postponed to the next sprint and blocked on the result of the discussion. Since we are working for a paying client we want to avoid spending time developing a solution that would end up being rejected by the module maintainers.

• To further assist the speedy acceptance of PRs, we assign members of our own team on doing code review on the upstream work. This meant we could already do a couple iterations and improve the code quality to a state that hopefully could be accepted right away, or only require minor modifications. Work with clear labels on the PRs to effectively communicate the state of the code. A label like “Ready for review” is very helpful and avoids people wasting time reviewing unfinished work.

• Provide extensive and well-documented tests on every PR. It cannot be underestimated how well a set of tests can help module maintainers to quickly understand the goal of a PR, and trust that the code does what it is supposed to do and will not break existing functionality.

An important rule to follow for making contributions to an open source project is to always put the requirements of the project first. It is tempting when working against a sprint deadline to propose a solution that provides some narrow functionality that your project needs, but often these are not aligned with the needs of the project’s wider use base. It is understandable in this case that a PR will be rejected. Often a Drupal module can be configured to cater to different use cases that are needed for different kinds of projects. This flexibility comes at the cost of added development time. One example in OG is the cleaning up of orphaned group content when a group is being deleted: for small sites this is done immediately, but for large sites OG offers an option to do this on a cron job.

As a developer working for a paying customer it can be difficult to justify why we are spending time to develop certain use cases which are not directly needed by the project, but are needed upstream. This means we need to pick our battles carefully. Often the additional work is not substantial, and will be compensated by the fact that the code will be maintained upstream or can be used by other projects within our organisation. But in other cases the work simply cannot be justified from the customer’s viewpoint. One example is the Organic Groups UI module. The way groups are created and managed in our project differed so much from the way this is done in OG that we decided to implement our group administration UI in our own custom module.

The decision to only work on what is strictly needed for the project also meant that we had to accept the fact that our project would probably go to production before the module would be completely finished. We mitigated this risk by implementing our own set of functional tests that fully cover our functionality and UI. We run these in addition to the test coverage of OG. This turned out to be a very good plan, since at the time of launch not only OG was still unstable, but also 20 of the other modules that we are using.

From our perspective the collaboration was a great success. We were able to contribute a significant part of the D8 port, the communication with the other OG contributors in the issue queue went super smooth, and we paved the way for using OG in our other websites. Since our project went live the nature of our contributions has changed – our grouping functionality is feature complete so we no longer need to make any sweeping changes, but we still discover bugs from time to time and were able to identify some performance bottlenecks. Along the way I also got promoted to co-maintainer, and whenever I am at a code sprint I make time to review PRs.

Maarten Segers

The first project where I used Organic Groups was a platform for the European Commission that enables learning and knowledge exchange through online groups where members can keep in touch with each other and share their work. At that time the platform was using Drupal 6 and we didn’t have a lot of the cool things we now have in Drupal 8, like configuration management, services and dependency injection, and the new and improved API’s for caching, entities, plugins, etc. I had used another module (Domain Access) on a different project before to solve the problem of having different subsites but ran into a lot of bugs related to the access layer and it was primarily used to support different domain names which we didn’t need for the EC platform.

What I liked, and still like, most about OG was that it was really well tested. If you make any changes to the default access system you’d better make sure you have tests for it because:

• Access API is one of the toughest API’s in Drupal
• Bugs with access can quickly become security issues (e.g. unauthorized access)

Developing this platform taught me that adding a “group” feature can quickly make your website become very complex from a functional perspective as it can affect other existing or new features. Let’s say we have a feature to show the latest news articles, once we have groups there might be a feature for:

• The latest news articles for a given group
• The latest news articles for your own groups
• The latest news articles for a given user’s groups

If we have comments there might be:

• Comments in a given group
• Comments in your own groups
• Comments for a given user’s groups

Now replace “comments” with “most recent comments” or “most active comments” and repeat. If we have a site search there will likely be a group search or filter and now imagine some more complex features like multilingual or content moderation and all the variations groups may introduce on them.

Before I met Amitai at DrupalCon Amsterdam in 2014, I wasn’t contributing very actively to Drupal but as Dries Buytaert puts it, I was a taker.

In my early days as a developer back in the ‘90s I used to build client server solutions and desktop applications. I built the software on Windows and Linux and assembled the hardware. Years later, I discovered these inexpensive microcomputers and microcontrollers that enabled me to build IoT solutions as a hobby. I’m still amazed how much more bang for the buck we get these days when it comes to hardware. But it wasn’t only the cheap hardware that enabled people to build IoT solutions. In the 90s most of the software was still proprietary. Although I did use sourceforge (again, as a taker), it wasn’t until I discovered GitHub that I really started contributing myself. I couldn’t believe how much was already out there, if you need something, anything, someone had probably already written a library for it. Also the ease of use and the low threshold to add or change a line of code made me become very active in contributing to a lot of different projects related to IoT libraries. Note that a lot of commits for bug fixes are one liners and a lot of PRs for new features on a well maintained project are just a few lines of code.

When Amitai asked me why I wasn’t so visible on drupal.org, I did some reflection and it was mostly because of two reasons. First, I didn’t like to attach .patch files to issues in the issue queue. I still don’t like it but now I try to convince myself that it’s worth it and I hope that the GitLab integration will arrive soon! In the meanwhile I’ve been contributing the most on modules that also live on GitHub like OG and Search api solr. The second reason was that most of our solutions were “tailored fit” custom modules.

Once you start contributing to open source project you quickly start to understand how in turn you benefit from it. I learned that by open sourcing it’s possible to reduce the total cost of ownership of a project. I’m not going too much into detail here on the advantages of doing open source right but I’ll list a few:

• with public scrutiny, others can identify and fix possible issues for you
• you learn a lot
• you may receive new features
• your own contributions may be further improved
• you avoid reinventing the wheel

In my opinion collaboration tools like GitHub have drastically improved open source workflows over the years. Did you know that you can now put multi-line comments or inline comments? Don’t hesitate to open an issue, a pull request on GitHub or join the Organic Groups Slack channel if you want to get updates have questions or you want to contribute yourself, it will be appreciated!

When developing the website for the European Year of Development (in Drupal 7) we used Organic Groups to model the countries and organisations. When designing the website’s features we made sure that the group feature didn’t have a huge impact on the complexity as we only had a few months to build. For instance, with Organic Groups it would have been possible to manage a hierarchical editorial workflow but instead we decided to use the existing platform to prepare and discuss the initial content.

When building websites using Drupal 6 and 7 I never really hesitated to use Organic Groups as the module had proved it was stable and well maintained. There were some performance issues but nothing we couldn’t handle. It wasn’t until we started building Drupal 8 sites that I started looking for alternatives as the Drupal 8 release for OG wasn’t in sight.

I met Kristiaan at DrupalCon in Dublin in 2016 and we discussed the Group module. By that time a lot of work had already gone into both the “Organic Groups” as well as the “Group” module. While both have a different architecture they try to solve issues in the same problem space. For example, each module has its own submodule to create group navigation (Group menu and OG menu).

A similar situation existed for the search problem space, more specifically for the Apache Solr integration in Drupal: there was the Apachesolr project and the Search api solr project. Both had a huge amount of submodules that tried to solve the same problems: Sorting search results, autocompletion, geospatial search, etc. This meant that two groups of developers were spending time to solve the same issues. The maintainers succeeded to join their efforts in Search api solr which also lead to a single Search API ecosystem.

Perhaps one day we can define one Group API as a basis for Drupal 10 and build our own Group ecosystem!

Kristiaan Van den Eynde

I’ve never really gone into the specifics of why I started developing Group and how I ended up in a position where I am invited to add my thoughts to a blog post written by three people working on a competing/complementing module. Maybe it’s time I cross that bridge publicly once and for all, because in my heart and mind I’ve already left that part of my past far behind me.

Way back in 2010, I started my first official IT job at a university college in Antwerp, Belgium. I had been tinkering with websites many years before that, but this time it was the real deal. After a round of headaches using closed source applications, we managed to convince the higher-ups to allow us to use Drupal for our websites.

One of the first Drupal 7 projects we built was a student platform which needed the functionality people have grown to expect from OG or Group, except there was only OG at the time. Having just had my first taste of what Drupal 7 development was like, I was still in this phase where everything is both daunting and complicated, but the solutions offered almost never seem to suit your needs. OG was not an exception to this rule.

This often triggered the naive, energetic caffeine junky that I was at the time to do all the things maintainers hate to see from their users: I complained in issues about how broken things were, about patches not being accepted, about the general approach of the module and so on.

And thus came the fateful day, where after a round of “frustrating” issues I encountered with OG, I went to see my boss and pitched him the idea that I could write this better and more tailored towards our needs. The fact that he was a very understanding boss and we had way more time on our hands than we needed led him to say yes, and in a single instant changed my career in the most unimaginable way ever.

When I initially started Group development, my main focus wasn’t to reinvent the functionality OG offered but rather to come up with a different data architecture, and as a result, a different developer and user experience. I must have struck gold somehow, because over the next few months of development, I got a lot of positive feedback and was therefore even more motivated to spend my every waking moment polishing my module.

Over the years to come, I went overboard in my enthusiasm and added far more features to Group than I cared to maintain. I was starstruck by the sudden popularity and wanted to appease everyone to make sure I kept that momentum going. This was a really bad idea.

Around the end of 2015, the fact that I was the maintainer of Group landed me a job at a prestigious Drupal agency in the UK: Deeson. My personal life had seen a few drastic changes which meant I no longer had all the spare time in the world to work on Group. So I brokered a deal where I got to work one paid day a week on Group. This is when I was first encouraged to begin development on the Drupal 8 version of Group and to start speaking at events.

I have since grown a lot both as a developer and a person and learned a few valuable lessons, the most important being to respect anyone willing to invest such a huge amount of time into writing a module the size of OG or Group. Next on that list is to first try to collaborate on a project before deciding to reinvent the wheel. Even if I have my career to thank for it, writing Group from scratch was an undertaking that should have definitely crashed and burned, yet somehow didn’t.

So here we are: A rookie turned senior who has come to respect and like Amitai, even if many years ago I strongly disliked his product. Talking to Amitai now, I realize that we both share the same knowledge and that we both want to fix this problem space the best way possible.

It is my hope that one day we can combine OG and Group into a single module that uses the best from both worlds and can be as well written and tested as Commerce. While that ship has definitely sailed for Drupal 8 and we might still have some minor disagreements about how to best approach certain functionalities, I hope to sit down with Amitai, Pieter and Maarten one day to make this happen for Drupal 9 or, more realistically, Drupal 10. In the meantime, I’ll just keep spending my one day a week (now at Factorial) to work on Group and Core.

And who knows: If us “burying the hatchet” someday leads to a single module whose approach people tend to disagree with over time, we might see another person like me step up and try to do it differently. I would certainly welcome the competition as, in the end, it improves the product for both parties involved. I would like to leave that person one piece of advice, though: Do not get dragged down the rabbit hole, keep a healthy work-life balance and try to respect the people who came before you. I know I should have.

For those wondering how "competing" module maintainers (og and group) get along in #Drupal.#DrupalCon pic.twitter.com/rCXJkW97dL

— Amitai Burstein (@amitaibu) September 27, 2016