# New ratings system for problem solving on Brilliant

So it turns out that Christopher J has predicted the future! We liked his idea so much we have been developing it for a few months now.

We have indeed just launched an adaptive ratings system on Brilliant. Levels still exist, except now instead of being determined by your performance over the past two weeks, your level is determined by your rating, which changes each time you solve or fail to solve a problem.

Read about levels and the new rating system here. You will note that the colors designating each level have changed as well. This was done to better capture the fact that levels on Brilliant are no longer sharply distinct things, but are more related along a continuum.

You can also link to an explanation of ratings, by clicking the "what's this?" link next to your rating on the problem solving page.

Remember while your levels are visible to everyone your rating in any given topic is visible only to you.

We hope this will help you gain more specific insight into your progress and how hard a particular problem is relative to your experience level. It will also help us better gauge the difficulty of the problems we distribute to you.

Let us know if you have any questions in this thread.

Note by Peter Taylor
5 years ago

MarkdownAppears as
*italics* or _italics_ italics
**bold** or __bold__ bold
- bulleted- list
• bulleted
• list
1. numbered2. list
1. numbered
2. list
Note: you must add a full line of space before and after lists for them to show up correctly
paragraph 1paragraph 2

paragraph 1

paragraph 2

[example link](https://brilliant.org)example link
> This is a quote
This is a quote
    # I indented these lines
# 4 spaces, and now they show
# up as a code block.

print "hello world"
# I indented these lines
# 4 spaces, and now they show
# up as a code block.

print "hello world"
MathAppears as
Remember to wrap math in $$...$$ or $...$ to ensure proper formatting.
2 \times 3 $$2 \times 3$$
2^{34} $$2^{34}$$
a_{i-1} $$a_{i-1}$$
\frac{2}{3} $$\frac{2}{3}$$
\sqrt{2} $$\sqrt{2}$$
\sum_{i=1}^3 $$\sum_{i=1}^3$$
\sin \theta $$\sin \theta$$
\boxed{123} $$\boxed{123}$$

Sort by:

It's always nice when reality conforms to our wishes. :)

- 5 years ago

Why do I see a dim star beside my rating for each of Maths and Physics ?? How do I enLIGHTen it ...

- 5 years ago

I think the new rating system is great but I think that the colours are not ideal. Now it is less easy to easily distinguish one level from another. Also, as previously voiced by other Brilliant members, the members who have ordered their coloured lanyards to indicate that they have reached a certain level have now been 'outdated' and no long correspond to the correct colour. I know that you mentioned that you did this to make the levels more like a 'continuum' which is understandable but I believe that the sharp and distinct colours such as blue, green, red and yellow make it clear which level you are in and cause for less confusion. Maybe having distinct colours will also make people more motivated to reach the next tier of levels. Someone would probably be more excited to go from yellow to green than to go from purple to a slightly darker shade of purple. I guess it doesn't really matter too much but I have seen many people such as myself on the discussion board that dislike the new colour scheme and it seems to be a common concern. Thanks for the update nonetheless as I really liked everything else that came with it.

- 5 years ago

I think that, rather, the colors should be a direct function of your rating, not your level, maybe

- 5 years ago

It would be like a rainbow! yay, cool idea.

- 5 years ago

hi， cole

- 5 years ago

Any estimates for rating comparisons? For example, what rating would roughly correspond to USAJMO? IMO? Is the scale similar to chess ELO?

- 5 years ago

One thing I noticed just now.. Some of the problems have higher ratings but lower points. How is this possible? One should get more points by solving harder problems.. right? Or is it that ratings of the problems are also determined by how many people and of what rating could solve these? If that is the case, then it is fine I think.

- 5 years ago

Points and ratings are not currently computed in the same way. This will be changing soon as well.

Staff - 5 years ago

Woww.. this is really cool!! This is very similar to rating system for chess which I always love.. :)

- 5 years ago

Will we level down if we don't do the problems?

- 5 years ago

I checked this today by activating Algebra, and then I viewed all four problems one after the other. My rating changed each time. (It started off mid-2100s for the first problem, and was down to 2094 by the last one).

Actually came here to query why the rating was changing so quickly, but realized after reading these posts that it's leveling me down just for viewing them.

Maybe one solution to the problem Calvin mentioned of exploitation is that if you solve at least one problem in the set for the week, then you do not get any rating loss for other problems in the same set that you viewed but didn't attempt?

- 5 years ago

Inactivity will not level you down. If you view lots of problems without solving any of them, you will indeed level down, however you can always regain your rating and your level by solving the problems you viewed.

Staff - 5 years ago

I do not think that viewing problems should make your rating go down. I often look at problems, knowing I won't have time to do them, just because they are interesting. Viewing problems and not solving them does not imply you can't solve them.

Also, I think you should be able to see the rating for problems that aren't in your own set.

Other than that, I like the new system, and how you can see the difficulty of problems. One question though: How are ratings calculated?

- 5 years ago

Yes.. I request Brilliant to change this.. Otherwise I will be afraid of viewing the problems in the first place. :(

- 5 years ago

I agree completely, the past month I just didn't look at the sets even though I know I would be missing out since I don't want to level down just because I don't have time really to do the problems. If someone can't do the problem, then it'd be obvious when they put in an answer and get it wrong. The problem with this is that it does not account for the fact when somebody doesn't get a problem so much that they don't even put in an answer. However, I think this is a lesser externality than having people not want to even look at problems in fear of leveling down.

- 5 years ago

Thanks for the feedback, Daniel. We are still thinking about the right way to handle "views" with no tries. Remember, though, that your rating is private. There is no shame in your rating going down a little.

Also, ratings is off in "Practice"—try problems with no fear!

Staff - 5 years ago

I also think we shouldn't lose points for viewing problems. And there's a difference between Practice and Challenges.

- 5 years ago

You can always wait until the next week and look at them. :) Just stop for one week, next week view them, won't decrease anything.

- 5 years ago

This is not a true statement.

Can you let me know why you thought this is true? If we stated it somewhere, I'd have to correct it.

Staff - 5 years ago

Well, after a week, you can see the answers and solutions, so it makes no sense for your rating to go down.

- 5 years ago

You only get to see answers and solutions to problems that you have already viewed the previous week, which already had the initial deduction (just once). Otherwise, we would not list any problems for the week under past solutions.

If you click on a public link (though a profile, or otherwise) to a problem that you have not seen before, then that would still affect your rating.

Staff - 5 years ago

Oh okay I did not know that. I stand corrected.

- 5 years ago

Peter, either you've forgotten to change the FAQ, or what you said above isn't true.

Quote from FAQ: "Don't worry; viewing and not answering problems will not cause you to level down, nor will taking a break from Brilliant."

- 5 years ago

Aha! Good catch. We forgot to update the FAQ.

That sentence is now half-true. Taking a break from Brilliant still won't level you down, but viewing enough problems, without solving them could level you down.

Staff - 5 years ago

Till today I was thinking that we are only levelled down on viewing sufficient problems and attempting sufficient number of them wrongly. Calvin, himself addressed this in a few discussions! So, is this part of the new system?

- 5 years ago

As we change the system, we have to update some of the rules, and resolve various issues that arise. Our mentality remains the same, and ratings allows us to be more flexible in responding to various scenarios.

A strong student in his level will not drop a level by glancing at 5-8 problems over 2 weeks. However, if someone viewed 10+ problems without being able to answer any of them, we believe it would be better to work on the fundamentals first. The system that you mentioned was abused by members who unable to answer any of the questions, but artificially maintained their level due to not attempting sufficient number of them wrongly each week.

Staff - 5 years ago

As I have already mentioned somewhere else, I think if users are allowed to level down own their own (but not level up of course!) then this issue will be solved quite easily. Why not to do that?

- 5 years ago

With the current ratings system, it is clear that anyone can easily (manually) level down by viewing enough problems / getting them wrong (as opposed to just stating it in the FAQ). They do not need to wait 2 or more weeks.

Staff - 5 years ago

I think users should have the option to manually level down if they want to, instead of arranging it by getting hard questions wrong on purpose.

- 5 years ago

Hmm.. I have one more question.. if I view the same problem more than once.. let us say I close my browser and reopen, then will my rating reduce every time?

- 5 years ago

Nope. It only decreases once (during the initial view).

Staff - 5 years ago

I would like that!

- 5 years ago

That's fair enough for me. I was expecting a much more stringent rule.

But, I request you to create a page listing all such rules and guidelines of the site and updating them whenever necessary.

- 5 years ago

A suggestion: shouldn't our rating increase when we publish a solution to a problem? Publishing, or even writing out a solution verifies that you were able to solve the problem without relying on things like computers for symbolic manipulation or anything of that sort, and are able to communicate your method clearly. (Yes, I know there are a couple of solutions out there that list programs to solve a problem, but those are in a very small minority.)

- 5 years ago

The problem is that then a lot of people would post poorly-written and erroneous solutions. Alternatively, upvotes could increase rating.

- 5 years ago

That works too.

- 5 years ago

Its really amazing! I dont know in which game it is used but I know that brilliant.org would be the first site to do this. HATS OFF TO BRILLIANT!!!!!!

- 5 years ago

Great improvement. I was wondering, does my rating increase more if I solve a problem with a higher rating? I suppose yes, and if so, does my rating increase even more if a problem turns out to be harder than at the time I solved it?

- 5 years ago

Yes. Solving a higher rated problem increase your rating by a larger extent, as stated in "what's this".

No. Your rating is only affected at the point when you (initially) viewed or answered the problem (both correctly and incorrectly). Viewing the question repeatedly (whether you solved it or not) will not affect your rating.

Staff - 5 years ago

Comment deleted Nov 03, 2013

Oh sorry, I misinterpreted your question. Your rating doesn't change in the future, from actions in the past.

Staff - 5 years ago

Okay! Yeah, that makes more sense. Though, I have noticed some problems having a very high rating just at the start, when nobody's solved it yet. That would mean that solving problems very early during the week could result in a rapid rating increase?

- 5 years ago

There are (will always be) edge cases in any such system. Several factors are involved, and we try and account for them.

Problem ratings at the start of the week are extremely volatile, since we have merely set an estimate, and are letting it stabilize to it's own rating. If we are uncertain about the rating of a problem, then it would have a smaller effect on your rating.

Staff - 5 years ago

You could calculate people's rating change after a few days, when the problems rating is settled, or even at the end of the week.

That way everyone will be treated equally when dealing with rating changes.

- 5 years ago

When is your level updated to match your rating? From my ratings, I should be a Level 4 in Physics and a Level 3 in Combinatorics, but instead I'm at Levels 3 and 2 respectively.

And why do the problem ratings typically fall outside the band for the level? For example, this week's Level 3 Algebra problem ratings are (at the time of writing) 1934 (Level 4), 1696 (Level 3), 2151 (Level 5!) and 2225 (also Level 5!). Only one of them falls within the band for Level 3!

- 5 years ago

Hi John,

When is your level updated to match your rating? From my ratings, I should be a Level 4 in Physics and a Level 3 in Combinatorics, but instead I'm at Levels 3 and 2 respectively.

Sounds like a bug! We'll check it out, thanks for writing.

And why do the problem ratings typically fall outside the band for the level? For example, this week's Level 3 Algebra problem ratings are (at the time of writing) 1934 (Level 4), 1696 (Level 3), 2151 (Level 5!) and 2225 (also Level 5!). Only one of them falls within the band for Level 3!

We are still tuning the ratings system. Note that a problem's rating is not necessarily supposed to line up with the user levels.

The important question for you is: do those problems feel too hard for you?

Staff - 5 years ago

I found another BUG here which I exploited once ...

In a physics problem I had used up 2 tries guessing the answers as A and B and I knew that my answer could be either of X or Y but because of only one try left I thought of the following...

I opened the same problem page in 2 tabs and gave the answer as X in one and as Y in the other and pressed enter in both almost at the same time .. Both pages showed that all my answers were wrong but also something else one showed that I had entered A,B & X the other showed that I had entered A,B & Y ... So I actually had used 4 tries and couldn't get to the answer.. I hope you fix this ;-)

- 5 years ago

The "Bug" is that the server can handle more than one request at the same time. While this may make it seem like you get 2 tries, whichever came in slightly after the other (from the perspective of the servers) would end up over-writing the other so it would basically be like tossing a coin between the two.

It is always a better idea to enter just the one answer that you think is more likely to be correct.

Staff - 5 years ago

Hmm...all my levels were upgraded accordingly when the new week started, except Physics. Rating-wise, I should be Level 4, but instead I am still Level 3.

- 5 years ago

According to your profile, you are now Level 4 in Physics!

Staff - 5 years ago

But the Challenge page still says I'm Level 3, and I got the Level 3 problems this week...

- 5 years ago

Yep. Your problems for the week are decided by your level at the beginning of the week. Next week, you will receive the Level 4 problem set.

Staff - 5 years ago

Ah, I see. Thanks for clarifying.

- 5 years ago

Well, they were OK for me. But my main problem was really with the disparity between rating and level. But since you lot will be tuning the ratings, I'll just wait and see what happens. Thanks for responding.

- 5 years ago

I don't think there is a level 6 or 120...

- 5 years ago

How can I know my rating?

- 5 years ago

Right now, the place to see your current rating in a topic is next to any weekly problem.

Staff - 5 years ago

- 5 years ago

Soon!

Staff - 5 years ago

I think you could add Selling Tips because when people do not know where to start, Only with Shopping Tip the Person has a Greater Sense about What Problem Want !

- 5 years ago

I've noticed that over the past few days my physics rating has been fluctuating even though I haven't looked at or attempted a problem since the beginning of the week. The last time I finished with physics for the week, my rating was slightly above 1500, about 1503. Then by yesterday it had gone down to 1497 and today it was gone back up to 1499. Why would it be changing like this?

- 5 years ago

Why is that the ratings of problems change so fast ?

- 5 years ago

If we have the rating 1750-2000, doesn't it imply that we are in the 5th level?

- 5 years ago

We have shown your rating on the challenges page, along with the next level up target.

If your rating is in the range 1750-2000, then you would be in level 4. There is a slight allowance for level 5 users who had a small drop in rating, due to getting a problem wrong or viewing more problems. So it's possible to have a rating just under 2000, while still be in level 5.

If you look at the levels page, we stated level up targets, which are nicely rounded off to 1250, 1500, 1750, 2000.

Staff - 5 years ago

Yesterday, it showed "level up at 1750" when I was in level 4 in number theory. Hence my question.

It is resolved now.

- 5 years ago

I have a complaint. That is every time the server updates, my rating goes down. And it has happened twice now. First I had 1731 for number theory, then went down to 1681 and now 1641. It seems it won't be long before i level down to level two. This has happened with all the other topics too. I was meant to be on level three number theory, algebra and number theory. It will be good if you could help me with it.

- 5 years ago

If you read the "What's this" explanation, your rating goes down when you initially view the problem, or answer it wrongly. It goes up when you answer the problem correctly. I've reviewed your rating changes, and they are occurring as intended by our algorithm.

Staff - 5 years ago

Ok, I understand, thanks for your help.

- 5 years ago

Can I get degrade immediately in Number Theory?

- 5 years ago

My Algebra has just been upgraded to Level 5. However, my rating is [removed - Calvin]

Shouldn't my Algebra be at Level 4? I mean, I like having my algebra rated highly, and personally I think I'm good at it (my favourite), but Algebra Level 5 is 2000 and above, right?

Just pointing out a possible error in the system

- 5 years ago

I see that your rating did cross over 2000 when you solved a probelm, then came down slightly when you viewed a new problem.

There is slight leeway in lowering a level, just so that you do not aggressively toggle back and forth between levels when you are right at the threshold.

Staff - 5 years ago

There is another issue with the "viewing penalty" namely viewing problems that are in other problem sets, like a level 5 combinatorics set for instances. When I was looking at a level 5 Number Theory problem the systems took off rating points.

- 5 years ago

How can you see level 5 problem if you are on level 4 (or that matter any lower level) ?

- 5 years ago

Go on the profile's of those in the respective level you want to observe, check their activity and you'll see the problems they have solved, e.g. the ones in that level.

- 5 years ago

A question and clarification. Do the ratings fall when you do not attempt any question? As it turns out that i had scaled the barrier from one level to another but when the new week came my ratings had fallen.

- 5 years ago

If you view the problem i.e. open it's link, your rating would fall, even if you didn't try to answer. If you get it wrong, your rating would fall even more. If you get it right your ratings would increase significantly enough to compensate for both reductions.

If you do not open problems link and thus don't see the problem, your ratings would not be affected.

- 5 years ago

Just had an idea: what if, instead of having problems by level, you could have problems by rating? I mean this in the sense that you wouldn't get "Level 4 algebra problems" but "Rating 1600-2100" problems, or something like that.

- 5 years ago

The rating system is simply awesome! Nice work brilliant team. A question i have in my mind- how much my rating increase whenever I solve a problem?

- 5 years ago

Love the new rating system. I think it is great!

This question is a bit off topic, but I was wondering why there are no more geometry questions like there used to be? And are they going to be brought back?

- 5 years ago

Also I can't access the Data Structures and Algorithms questions. Is this the same for everyone or is it a bug?

- 5 years ago

Same for everybody.. it will come back soon..

- 5 years ago

Ok. Thank you :)

- 5 years ago

Why have I gotten so many vote downs? :(

- 5 years ago

Probably because your question was not on topic, and they have been asked many times before.

- 5 years ago

Hi. I just saw some comments about the "views" and I got an idea, so I want to share it. Maybe, if the problems are attached in a PDF document, the users will get able to see them without starting the weekly challenge. That could be great, because then the rating will be evaluated correctly. If a user opens the PDF just because he/she wants to see the problems, he or she will not be penalized. But if he/she starts the challenge, it will mean that he/she knows some of the problems, so in case of missing them, the user can be penalized justly. How does it looks?

What's the difference between this and just taking away the viewing penalty?

- 5 years ago

Well.. you'll be able to decide if whether you are prepared or not. I think the penalty is justified after you miss a problem, and it is possible only if you see it in other way, different than starting the challenge. It's just an idea...

If one is not prepared one should be penalized and later sent in down level. You can see problems one by one while solving each before viewing the next. That way you only lose for actually not doing the problems instead of just not having time.

Anyhow Brilliant has a pressure of maintaining sufficient traffic of students actually solving problems, so this makes lot of sense from their perspective.

- 5 years ago

With prepared i meant like being prepared to start working with the problem on that specific moment.

Well you have entire week :) Your ratings would be deducted only once for viewing. You may few it thousand times (or more) after that without any loss.

- 5 years ago

Due to this new ratings system, I've been leveled down to level 4 in algebra. Now, I'm pretty darn good at algebra. In fact, I just finished all 4 algebra level 4 problems this week in ≤ 15 minutes. Please level me back up to level 5 in algebra.

- 5 years ago

I see that you are now level 5 in Algebra. If you believe that this is still an issue, send me an email.

Note that your ratings will move around some during the week, as you work to solve problems.

Staff - 5 years ago

×