×

# A Discussion of the Brilliant Level System

After posting $$13$$ problems to Brilliant.org, I've finally created my first $$400$$ point problem! The reason that I'm posting this is that it definitely should not have been the first.

I don't mean to promote my problem, but I suggest that anyone reading this should try out this problem that I posted about a week ago. It may seem unassuming at first, but I assure you that once you try it you'll shortly realize that this problem isn't for the faint of heart. If I may quote fellow Brilliant user Mr. Pi Han Goh, when he read my solution to the problem he stated "Holy Crap! This is soooooooooooooo long."

After writing this problem, I expected it to easily be a Level $$5$$, worth something in the high $$300$$s, but after a few days and $$82$$ views, I was surprised to find that it's only been rated a Level $$4$$, worth a measly $$160$$ points.

I completely understand why it's currently rated this way. Brilliant.org rates their users and their questions based on a variation of the ELO rating system. For those of you who are unfamiliar with this system, it's a self-adjusting system of rating people (and in this case problems) based upon their interaction with other people with ratings. The way it works is that everyone is given a score to begin with. Those with higher scores are deemed "better" and those with lower scores are deemed "worse", comparatively. The system assumes that a person with a higher score is going to win if matched up against someone with a lower score. Each match affects the score of the two people involved. If two people are evenly matched, the scores of each person won't change that much after the match. If the person with the higher score wins, then that person's score goes up a little, and the person who lost goes down a little. However, if the person with the lower score wins, this is considered an "upset" and the weaker person's score will go way up, while the supposed "better" person's score will go way down.

Brilliant's users and problems follow a similar system. If a person who has a low ranking solves a really hard problem, then that person's ranking will skyrocket, while that problem's ranking will plummet. Likewise, if a person of a high rank can't solve a problem, that problem's score is likely to increase. The system really is ingenious, and seems to work beautifully most of the time. As you can probably guess, I would like to talk about the times where it doesn't work.

When I wanted to post the above problem, I happened to be ranked Level $$4$$ in Combinatorics, not because I couldn't solve tougher problems but because I hadn't solved enough yet to get my rank there. Brilliant prevents you from posting problems above your level of aptitude, therefore the highest level I could post the problem at was Level $$4$$, which I thought was fine because eventually the ELO system would bring my problem up to the level that I thought it should be at. I thought that enough people would try the problem (and fail) that the problem's level would eventually raise to the level that I expected the problem to be at. However, after $$82$$ views, the problem only has had $$3$$ attempts, and this got me curious.

Why were people only viewing my problem and not trying to solve it? Did people try to work out the problem and then realize it was too hard and never tried to submit an answer? I personally think that that's probably unlikely. While not everyone may feel this way, I know that if I start working on a problem, I can't sleep knowing that I don't know how to solve a problem. If I don't get the problem the first time through I'll make good use of those $$2$$nd and $$3$$rd tries, and if by then I haven't gotten it right I'll scour the solution page looking for ways to solve the problem that I hadn't thought of, in case a problem like it comes across my feed again. So if that's not the reason, what could it be? My best guess? The problem isn't worth their time.

When you read solve a $$400$$ point problem, there's a sense of pride that comes with it. It feels good to know that you just solved one of the hardest problems on Brilliant.org. You're part of an elite few, the upper echelon of a certain branch of mathematics. What would you rather solve, a tough problem worth $$400$$ points or one only worth $$160$$? Someone who reads my problem is probably thinking "I could spend all day on this problem and get $$160$$ points, or I could knock out $$5$$ $$200$$ point questions in half an hour." If my problem isn't being attempted, then the ELO system can't be put into action, and so my problem sits, waiting to be attempted, ranked lower than it should be.

My question to you is, what are some changes that can be made to Brilliant's leveling system that can account for these kind of erroneous appropriations? Should the so-called "level cap" on posting problems be eliminated? What would be the implications of doing so? Is there a way of asking users who have attempted the problem whether they thought the level of the problem was appropriate? How can we get a better assessment of a problem's difficulty right from the get-go? Please feel free to leave a comment so we can discuss this issue together.

EDIT: My problem has since been simplified so that it's no longer as tedious to solve as it once was. It's now at Level $$5$$ hovering around $$350$$ points. Thanks Brilliant.org users for your helping me fix this issue!

Note by Garrett Clarke
2 years ago

Sort by:

Point 0 - Improved feedback from members about problems

We have been working on this, through a variety of ways. Because we are breaking new ground, we have to be careful with introduced changes, and how the community responds to it. For example, at the start of the year, we introduced a way for members to directly get feedback when they report a problem. The community has responded favorably to this (after making some tweaks), and we are looking at updating this system, by allowing members (not just moderators) to suggest changes to the problem’s statement, topic and even level.

Having said that, let me address some of the issues that you brought up. Staff · 2 years ago

@Michael Mendrin (in response to your comment in point 3) Right, that's why I'm really excited about this. Just imagine if our percentage of great problems went up from 10% to 30%, that would be so amazing. Sometimes, it just takes tiny changes (like spelling, grammar, Latex) to significantly raise the quality of a problem. Staff · 2 years ago

Calvin, one of the difficult challenges Brilliant.org faces, I think, is how to find the right balance between "interesting, hard problems", and problems that most people can solve. It's become obvious to me that the popularity of a problem is almost inversely proportional to the difficulty of it. Sure, I can post problems that are extremely hard, but then nobody wants to try them, and hurts Brilliant.org participation rate. The magic number seems to be around 200 to 300--hard enough to get people excited about attempting them, but not so hard that eyes just glass over. · 2 years ago

But that's such a fun challenge to solve :) Certainly better than the issues that some schools face with "How do I teach my students mathematics".

Yes, understandably, the problems that are more apparently popular, are those that are easier to understand and solve. That is just a fact of life, it's hard to get people to care about something that they do not understand.

The approach we are taking, is to find a group of people who enjoy having their eyes glass over, and then break the problem apart to figure it out. To a certain extent, this is already happening at the lower levels. It is harder to come across at the higher levels, in part due to the difficulty of the material and the length of time that is required. E.g. Even if a problem inspires some one to learn about Galois theory over the period of a month, they would be less likely to comment back on the original problem. Staff · 2 years ago

Ah, yes, exactly, that is another problem. Suppose a problem involves Galois theory (the note about Bravais lattices in 3D comes into mind), by the time I get around to answering it, it's long gone from the scene. So, it ends up just being a conversation with the author, and you hope he's/she's still around to comment. Likewise if I'm the author of the problem.

Hey, Brilliant.org is work in progress. · 2 years ago

Hi Garrett,

Thanks for your feedback, it is greatly appreciated. We do make tweaks to the level system, and feedback like yours help us understand what improvements there are to make. We have to manage several conflicting wants and desires, and so such concerns cannot be taken in isolation, and we do have to consider rippling effects.

I have broken up my reply into several different comments, so that people can reply to the appropriate part. If there are any other issues that you would like me to address, please let me know. Staff · 2 years ago

I wanted to ask what is the real worth of a problem -1)Level set by a moderator or 2)Level secured by problem on its own since set by user without setting up by mod.1) or 2)? · 1 year, 12 months ago

The real worth of a problem is what you get out of it, through the effort that you put in for solving the problem of understanding the solution. Staff · 1 year, 12 months ago

Point 2 - Determination of problem ratings

In an idealized world, I would love to be able to review every single problem that comes in, to ensure that it has an accurate rating attached to it. However, we live in a world where Calvin Lin only has 24 hours a day (that extra leap second hasn’t helped much), and also has numerous demands on his time.

We have gotten more aggressive with releasing ratings earlier (IE going from level pending (100 points) to level X). We feel that this is beneficial both to the problem creator and the community. Unfortunately, it also does mean that abuses / edge cases of the system are more likely to surface before they are corrected.

I think that we are close to a sweet spot of releasing accurate ratings for a high proportion of problems. I do not think that major changes in withholding / releasing problem ratings will be beneficial. Staff · 2 years ago

Are there stats on how many problems are posted to Brilliant per hour or per day? · 2 years ago

Calvin, I'd like to make a note about something that I find somewhat unusual. Several times now, I've posted a Level 5 problem, and there's a lot of attempts without any success, and suddenly I start getting people that have solved it, and they have no history of solving problems anywhere near that level, and have often recently or just joined Brilliant (like it's the first problem they've tackled and solved), and they then continue to only tackle low level problems. Any suggestions what might be going on?

If this happened once or twice, I would have said, "well, that can happen, maybe". But after four, five, six or more times, out of less than 3 dozen Level 5 problems I've posted, it does make me wonder. · 1 year, 12 months ago

There could be several reasons for that.

1) They are cheating accounts.
2) They came across the problem on Facebook (or another site), saw the answer in the comments, and then typed it in. However, they cannot actually solve the problem on their own.
3) They came across the problem, and solved it properly. However, our introductory experience leads them through easier problems, and so they end up stuck there. They do not figure out that they can filter for Level 5 problems, in part because we start everyone out at level 5.

In the case of 1, I can't be bothered with them.
In the case of 2, that is the behavior that they should have.
In the case of 3, we will be working on improving the experience for them.

Right now, the case of 3 is quite rare. Staff · 1 year, 12 months ago

I thought Brilliant.org can't cheat? Because every person has only 1 account. Every problems can't be solved until you wrong or get solutions. And, the Facebook problems's points are not so attractive to a person like me, I can totally solve it by myself. · 1 year, 11 months ago

Comment deleted Jul 23, 2015

But in the case of 1, it sure seems like a lot of work creating a new Brilliant account just to be able to say you've "solved" a hard problem? All it does is to mess up the ratings and impresses nobody. · 1 year, 12 months ago

Welcome to the intenetz. People do all kinds of strange things for all kinds of reasons.

It doesn't really mess up the ratings. New accounts have minimal effect on problem ratings (even new problems), because we're still trying to determine what level the accounts are at. Staff · 1 year, 12 months ago

To what extent does the % of solvers affect the level? I couldn't help but notice that my problems levels were completely messed up.

https://brilliant.org/profile/vishnu-mwwm5x/sets/combinatorics-problems/

Problem 1 which majority of people got was level 3 while problem 3 which only 13% of people got was level one. I'm not completely sure but I was level 2 in Combinatorics when I posted this, I think. · 2 years ago

The percentage of solvers plays a minor role (especially since your problem has been seen by < 100 people). What is more important is the composition of the solvers. For example, if all of the people who solved a problem were level 5, then that doesn't really give us much information about the problem. Whereas, if all level 3's who tried the problem can solve it, and all of the level 1's who tried the problem cannot solve it, then that tells us that the problem should be around level 2 (give or take). As such, the percentage of solvers is (mostly) irrelevant, if you do not know the underlying population.

Bear in mind that as a problem rating adjusts, it will also be filtered into different levels. So, it is not inconceivable the solver percentage could go down AND the rating could go down at the same time. This often applies to problems that are placed in too high a level, where the rating is going down as expected but because more low levels see it and can't solve it, the percentage decreases. Staff · 2 years ago

Point 1 - Level cap on problem creation

In an idealized perfect world where everyone understands everything and agrees with how things should be done, I agree that “No level cap on problem creation” should be the default behavior. However, if we were to proceed and do so, here is what would quickly happen:

1. Many people would (for various reasons) post a problem as level 5.
2. The level 5 filter will be flooded with problems that are across a huge range.
3. Level 5’s would complain that most problems they see are not correctly rated.
4. Brilliant level 5 would be meaningless to an informed observer.
5. Brilliant would be unable to attract or retain Level 5.

We already see steps 1-3 happening. Often in the solution comments, we see things like “I can’t believe this is a level 5 problem, it is so easy”. We have been doing our best to clam down on such instances by issuing a stern warning.

Because of that, I am willing to comprise and deal with the edge case of “new-ish member with low level posts a really hard problem”. The moderators do look into popular problems, and help to adjust the level if they feel strongly about it. It is not a perfect system, and problems with low views or attempts do fall through the cracks. Staff · 2 years ago

I 100% agree, the level cap must be maintained, as should the idea of what it means to be a Level $$5$$. Clearly, what makes one problem hard and another trivial is subjective. One person might take hours to solve a problem while another might see the solution in a matter of minutes. I personally have been tripped up by some Level $$2$$ problems that I thought were much harder than I had anticipated, and I've also had days where I'll be solving Level $$5$$s left and right, but it all has to do with the specific user's mathematical background. You recently commented on one of my problems saying that the Principle of Inclusion and Exclusion could be used to solve the problem. I had no idea what that was, so I clicked on the link you provided me and read all about and now I know how it can be used to solve the problem. It took me over an hour to verify my result before posting the problem, but for someone who knew that technique, I'm sure they would find this problem to be a simple feat. So what does it mean for a question to be a Level $$5$$? Is it a problem where a majority of people find it very difficult to solve? Or is it a problem that is meant to inspire the reader to not only use advanced techniques but to encourage them to explore the subject further? Just some things to think about. · 2 years ago

Point 3 - Problems with low views / attempts

As you pointed out, one drawback of the rating system, is that if a problem doesn’t “compete”, then the rating would not be updated. That is actually very good feedback for interested problem creators like you.

For problems with low views, why are people not interested in it? Is the problem just too boring and standard? Is there a way to attract people to click on it?

For problems with more views, but a low attempt rate, what is it that causes people to not attempt it? Is the problem way too long-winded and has so many words and statements that after a while, you wonder what the creator is thinking of and question your sanity for reading it and abhor the use of so many conjunctions and managed to consume your dinner while reading it and .... Most internet users would have disengaged already, so even as you pat yourself on your back for making it so far, be aware that not everyone else is as engaged with your problem as you would be.

For a “hard” problem, what makes it good isn’t that “Oh there are many numerous hoops that I make the solver jump through, just because I can”. As an explicit example, I don’t think your problem should involve $$\tan \sum \theta _ i$$. That would be a major reason to distract people from solving the rest of your problem. Just imagine if “After you solve this part of the problem, you have to go run 3 miles, swim half a mile and bike 12 miles”. If I had to do that, I would join a triathlon instead.

Lastly, for problems with a very low attempt / view rate, but an extremely high correct / attempt rate, that is often an indicator of an uninteresting problem that several people are cheating on. Of course, this is not a hard and fast rule, and there can be edge cases to consider. Even if the problem is truly hard, having the high correct rate does reduce the rating of the problem.

Note: While the base system that we use is Glicko, there have been numerous tweaks done to it, to account for various effects. Staff · 2 years ago

Wow. "As an explicit example, I don't think your problem should involve $$tan\sum { { \theta }_{ i } }$$. That would be a major reason to distract people from solving the rest of the problem." Right on the nose! After starting to read this note, I decided to tackle Garret's problem, and had it all worked out, until I got to the last part. It made me suspect that this could be a gotha question, and I didn't have to time to re-read everything and try to figure out all the ways I could be tricked. So, I let it pass.

I have a real bad habit of losing interest once I've solved a problem. · 2 years ago

You're absolutely right. After reading your comments, it occurred to me that I should probably change the question. To avoid having the answer changed, I could change the statement with $$\tan \sum \theta_i$$ to "If $$n$$ is the number of solutions, what is $$20n-1$$?" Do you think that the problem would benefit from this change or do you think that it's just too late and I should leave it as is? · 2 years ago

See Calvin Lin. I'm encouraged by all the effort Brilliant.org has made on feedback on "problems that have problems with them". Myself, whenever I see a problem that has potential to be a good one, I'll sometimes try to convince the author to make a few changes to it, to "fix" its shortcomings. The most common shortcoming is ambiguity, i.e., the answer depends on the reading of the problem, so that the solution isn't unique.

I'm going to wait until your chessboard trip problem is cleaned up a bit, before I put in my "correct answer". · 2 years ago

Certainly that would help improve this problem. If you change it, let me know what you want the answer updated to.

E.g. You cold change it to "What is n, state the value of 20n-1" for the time being, and then I can convert it to "what is n" and update the answers accordingly. Staff · 2 years ago

That's exactly what I did, thank you! @Michael Mendrin, I hope your eyes no longer feel the need to glass over! · 2 years ago

Just now put in my answer, and commented on the method of solution. It was a fun problem, once you got rid of the baffling trigs. · 2 years ago

Well thank you for saying so, and thanks for your comment! · 2 years ago

Very good point about my use of $$\tan \sum \theta_i$$, I can see how that could turn some people off. Just because a problem takes a long time to solve doesn't make it difficult, it just makes it tedious. I will say that I find it ironic that the view on this problem has gone up by $$40$$ since I posted this note, and the attempt count is still only $$3$$ haha · 2 years ago