🎉 Exercism Research is now launched. Help Exercism, help science and have some fun at research.exercism.io 🎉
News, interviews and articles from the Exercism community.

Was V2 worth the effort?

A couple of weeks ago, I posted an issue about our upcoming journey towards Exercism V3. Someone posted a comment saying that they hoped the transition would be smoother than V1 to V2, and questioning whether a lot of changes we made were worth the hassle. Both are excellent points to raise and show that we've clearly not explained the impact that the V2 changes did have and why we believe they were worth it. I thought it would, therefore, be worth writing a short post outlining the changes we saw in the data, and also the metrics that we're targeting with the push to V3.

(Three notes on the numbers. Firstly, the metrics below are roughly ordered by the flow of the site, rather than of importance. Secondly, I've used a 17-month window on either side of the launch of V2 for comparisons. When I say "V1", I mean the final 17 months of V1. There's no entirely accurate way to do this, as V1 had many small changes in the 17 months prior to launch, and V2 had substantial improvements over the period since it launched. We also have outlier days, like the one during V1 when GitHub featured us, and we had 30x traffic for a day or so. I've not worried about that and just stuck to a fixed window either side, believing it to be the fairest way to measure things. Finally, there are many metrics that I could have used for each point. I've chosen ones that I feel represent the story that the numbers tell as a whole, and that were easy for me to work out across historic and modern data).

V1 -> V2 Metrics

Visitors signing up (Branding)

One of our aims was to make Exercism feel more welcoming to people who might have been put off by the original (red + devil) branding. To many, this seemed like a minor change, because they weren't put off by the branding and so the effort seemed like a waste of time. However, we believed that Exercism didn't exude "welcoming, inclusive community" in V1, and it was important for us to put that right.

How many people signed up to Exercism over the periods, and what was this as a percentage of new visitors?

  • V1: 62,618 (17% of new visitors sign up)
  • V2: 188,844 (48% of new visitors sign up)

This first change was substantial. Our V2 landing page converts at 48% as opposed to 17% for V1. That's pretty much purely down to brand and clarity over what we do and who we are. So I strongly believe that the effort that went into rebranding Exercism was worth the effort.

One caveat is that V2 probably encourages people to sign up who aren't yet ready for Exercism as it stands. V2 isn't set up for total beginners, yet the landing page for V2 is much more welcoming for total beginners, so we probably get a lot of users who aren't yet ready for Exercism. One way to account for this would be to consider not only the number of visitors signing up but rather how many visitors signed up and managed to submit a solution. For this, we see:

  • V1: 26,194 (7% new visitors sign up and submit)
  • V2: 52,533 (13.9% new visitors sign up and submit)

So from this, we can say that the branding has accounted for approximately a doubling of our true "conversion rate", which makes the rebranding effort definitely worthwhile. It's also worth noting that as we move forward to being beginner-friendly (V3.x), this caveat will go away altogether.

Solutions getting feedback (Mentoring)

The second thing we wanted to improve was the amount of mentoring per solution. In V1, it was a bit of a lottery - you might get comments, or you might not, purely dependent on whether someone noticed your submission. A big part of the formal mentoring of V2 was about making this fairer, consistent, and reliable.

The metric I've used here is the percentage of solutions that receive a comment.

  • V1: 3.5% of all solutions
  • V2: 64% of solutions where mentoring has been requested (26% of all solutions)

This is the phenomenal statistic in my eyes and the big indicator that V2 took us in the right direction. You're now 18x more likely to receive feedback on your solution in V2 than in V1. There's a lot we got wrong with mentoring in V2 (see the next section!), but I feel hugely proud and grateful to our mentors for the fact that Exercism offers a place where if you want mentoring, you'll more likely than not receive it.

Stickiness

Finally, we wanted to improve "stickiness" - the amount of usage of the site. If people are enjoying Exercism and finding it useful, they'll do more with it. There are two sides to this, the number of times someone comes back to the site and the number of solutions someone uploads.

Returning users:

  • V1: 4.79 sessions / person
  • V2: 5.42 sessions / person

Volume of uploads:

  • V1: 8.0 solutions / person
  • V2: 8.2 solutions / person

Both held pretty much steady. This is one area where I hoped we'd see a big increase in V2, and we didn't. The wait times for mentoring, and the blocking mechanisms that stop progression without mentoring both made V2 a very slow experience. For someone wanting to quickly progress through a language, the huge mentor queues made it very difficult to learn at a reasonable enough rate. The experience here will be very track-dependent. For tracks with wait times longer than a day, the dropout rate is noticeable. For those with longer wait times (e.g., Rust, Python), the dropout rate is substantial.

The changes in V3 are largely aimed at fixing this problem - making Exercism a place where you can quickly learn or practice a language, but where you can also have helpful asynchronous conversations with mentors without having your progress blocked.

Moving towards V3.

For V3, I hope to see all three of these metrics continue to rise. My aim is that people hear about Exercism and sign up, that they manage to submit solutions and get deep into tracks without feeling frustrated, and that when they want mentoring, they receive it.

I hope that the statistics above demonstrate that we're moving in the right direction and that the work everyone has put into V2 has been worthwhile. There's going to be a lot of effort needed from the community to get us to V3, but I'm confident that when we take stock of Exercism 24 months from now, we will see similar increases and that all of the next phases of work will prove worthwhile. Exercism is our community - thank you all for everything you do.

Avatar of author
Jeremy Walker
Co-founder of Exercism. I'm an entrepreneur and software developer, and have been running a variety of businesses and non-for-profits for the last decade in the fields of medicine, education and artificial intelligence.
View Profile

Community Comments

Great to see the growth in Exercism due to v2 update and right now i excited to see whats is coming in v3! One think that could improve users sign up is the use of explainer video at the beginning of the website (motion design). Other think that could help is a walkthrough video for the setup of the cli and node.js

Just signed up a few days ago for the Golang track - the site is awesome! Thanks so much!

Following on from the comment before: Better onboarding would be a huge plus - I found it difficult to figure out where/what the CLI was, how it fit in to the exercises and then also how to submit (I didn't notice the 2nd block on the right hand side).

A bit of reading and research got me there, but a little more handholding or a quick explainer video on the first dummy exercise would have been great.

The other thing I would love to see added to the roadmap is possibly a premium subscription model. Premium users could have a priority mentoring queue - and the revenue generated could possibly be used to reward dedicated mentors (only get rewarded after being a mentor for x time, rewarded per mentored exercise).

Thanks for all the work tho :)

Thanks both of you. Better onboarding sounds like something we should definitely work on! :)

We won't be adding a paid Premium option as we don't want people to be able to get a better quality service based on their financial situation. But we may have alternative premium-style features based on non-financial values (e.g. contribute more, get higher priority mentoring, where "contributing" might be mentoring, contributions to exercises, or docs, etc).