Waste less time on Facebook — follow Brilliant.
×

New ratings system for problem solving on Brilliant

So it turns out that Christopher J has predicted the future! We liked his idea so much we have been developing it for a few months now.

We have indeed just launched an adaptive ratings system on Brilliant. Levels still exist, except now instead of being determined by your performance over the past two weeks, your level is determined by your rating, which changes each time you solve or fail to solve a problem.

Read about levels and the new rating system here. You will note that the colors designating each level have changed as well. This was done to better capture the fact that levels on Brilliant are no longer sharply distinct things, but are more related along a continuum.

You can also link to an explanation of ratings, by clicking the "what's this?" link next to your rating on the problem solving page.

Remember while your levels are visible to everyone your rating in any given topic is visible only to you.

We hope this will help you gain more specific insight into your progress and how hard a particular problem is relative to your experience level. It will also help us better gauge the difficulty of the problems we distribute to you.

Let us know if you have any questions in this thread.

Note by Peter Taylor
3 years, 9 months ago

No vote yet
67 votes

Comments

Sort by:

Top Newest

It's always nice when reality conforms to our wishes. :) Christopher Johnson · 3 years, 9 months ago

Log in to reply

Why do I see a dim star beside my rating for each of Maths and Physics ?? How do I enLIGHTen it ... Santanu Banerjee · 3 years, 9 months ago

Log in to reply

I think the new rating system is great but I think that the colours are not ideal. Now it is less easy to easily distinguish one level from another. Also, as previously voiced by other Brilliant members, the members who have ordered their coloured lanyards to indicate that they have reached a certain level have now been 'outdated' and no long correspond to the correct colour. I know that you mentioned that you did this to make the levels more like a 'continuum' which is understandable but I believe that the sharp and distinct colours such as blue, green, red and yellow make it clear which level you are in and cause for less confusion. Maybe having distinct colours will also make people more motivated to reach the next tier of levels. Someone would probably be more excited to go from yellow to green than to go from purple to a slightly darker shade of purple. I guess it doesn't really matter too much but I have seen many people such as myself on the discussion board that dislike the new colour scheme and it seems to be a common concern. Thanks for the update nonetheless as I really liked everything else that came with it. Cole Coupland · 3 years, 9 months ago

Log in to reply

@Cole Coupland I think that, rather, the colors should be a direct function of your rating, not your level, maybe Matthew Lipman · 3 years, 9 months ago

Log in to reply

@Matthew Lipman It would be like a rainbow! yay, cool idea. Justin Wong · 3 years, 9 months ago

Log in to reply

@Cole Coupland hi, cole Andy Qiu · 3 years, 9 months ago

Log in to reply

Any estimates for rating comparisons? For example, what rating would roughly correspond to USAJMO? IMO? Is the scale similar to chess ELO? Taylor Lau · 3 years, 9 months ago

Log in to reply

One thing I noticed just now.. Some of the problems have higher ratings but lower points. How is this possible? One should get more points by solving harder problems.. right? Or is it that ratings of the problems are also determined by how many people and of what rating could solve these? If that is the case, then it is fine I think. Snehal Shekatkar · 3 years, 9 months ago

Log in to reply

@Snehal Shekatkar Points and ratings are not currently computed in the same way. This will be changing soon as well. Silas Hundt Staff · 3 years, 9 months ago

Log in to reply

Woww.. this is really cool!! This is very similar to rating system for chess which I always love.. :) Snehal Shekatkar · 3 years, 9 months ago

Log in to reply

Will we level down if we don't do the problems? Wei Jie Tan · 3 years, 9 months ago

Log in to reply

@Wei Jie Tan I checked this today by activating Algebra, and then I viewed all four problems one after the other. My rating changed each time. (It started off mid-2100s for the first problem, and was down to 2094 by the last one).

Actually came here to query why the rating was changing so quickly, but realized after reading these posts that it's leveling me down just for viewing them.

Maybe one solution to the problem Calvin mentioned of exploitation is that if you solve at least one problem in the set for the week, then you do not get any rating loss for other problems in the same set that you viewed but didn't attempt? Matt McNabb · 3 years, 9 months ago

Log in to reply

@Wei Jie Tan Inactivity will not level you down. If you view lots of problems without solving any of them, you will indeed level down, however you can always regain your rating and your level by solving the problems you viewed. Peter Taylor Staff · 3 years, 9 months ago

Log in to reply

@Peter Taylor I do not think that viewing problems should make your rating go down. I often look at problems, knowing I won't have time to do them, just because they are interesting. Viewing problems and not solving them does not imply you can't solve them.

Also, I think you should be able to see the rating for problems that aren't in your own set.

Other than that, I like the new system, and how you can see the difficulty of problems. One question though: How are ratings calculated? Daniel Chiu · 3 years, 9 months ago

Log in to reply

@Daniel Chiu Yes.. I request Brilliant to change this.. Otherwise I will be afraid of viewing the problems in the first place. :( Snehal Shekatkar · 3 years, 9 months ago

Log in to reply

@Daniel Chiu I agree completely, the past month I just didn't look at the sets even though I know I would be missing out since I don't want to level down just because I don't have time really to do the problems. If someone can't do the problem, then it'd be obvious when they put in an answer and get it wrong. The problem with this is that it does not account for the fact when somebody doesn't get a problem so much that they don't even put in an answer. However, I think this is a lesser externality than having people not want to even look at problems in fear of leveling down. Michael Tong · 3 years, 9 months ago

Log in to reply

@Daniel Chiu Thanks for the feedback, Daniel. We are still thinking about the right way to handle "views" with no tries. Remember, though, that your rating is private. There is no shame in your rating going down a little.

Also, ratings is off in "Practice"—try problems with no fear! Silas Hundt Staff · 3 years, 9 months ago

Log in to reply

@Silas Hundt I also think we shouldn't lose points for viewing problems. And there's a difference between Practice and Challenges. Ryan Wood · 3 years, 9 months ago

Log in to reply

@Daniel Chiu You can always wait until the next week and look at them. :) Just stop for one week, next week view them, won't decrease anything. Yong See Foo · 3 years, 9 months ago

Log in to reply

@Yong See Foo This is not a true statement.

Can you let me know why you thought this is true? If we stated it somewhere, I'd have to correct it. Calvin Lin Staff · 3 years, 9 months ago

Log in to reply

@Calvin Lin Well, after a week, you can see the answers and solutions, so it makes no sense for your rating to go down. Daniel Chiu · 3 years, 9 months ago

Log in to reply

@Daniel Chiu You only get to see answers and solutions to problems that you have already viewed the previous week, which already had the initial deduction (just once). Otherwise, we would not list any problems for the week under past solutions.

If you click on a public link (though a profile, or otherwise) to a problem that you have not seen before, then that would still affect your rating. Calvin Lin Staff · 3 years, 9 months ago

Log in to reply

@Calvin Lin Oh okay I did not know that. I stand corrected. Daniel Chiu · 3 years, 9 months ago

Log in to reply

@Peter Taylor Peter, either you've forgotten to change the FAQ, or what you said above isn't true.

Quote from FAQ: "Don't worry; viewing and not answering problems will not cause you to level down, nor will taking a break from Brilliant." Ton De Moree · 3 years, 9 months ago

Log in to reply

@Ton De Moree Aha! Good catch. We forgot to update the FAQ.

That sentence is now half-true. Taking a break from Brilliant still won't level you down, but viewing enough problems, without solving them could level you down. Peter Taylor Staff · 3 years, 9 months ago

Log in to reply

@Peter Taylor Till today I was thinking that we are only levelled down on viewing sufficient problems and attempting sufficient number of them wrongly. Calvin, himself addressed this in a few discussions! So, is this part of the new system? Sambit Senapati · 3 years, 9 months ago

Log in to reply

@Sambit Senapati As we change the system, we have to update some of the rules, and resolve various issues that arise. Our mentality remains the same, and ratings allows us to be more flexible in responding to various scenarios.

A strong student in his level will not drop a level by glancing at 5-8 problems over 2 weeks. However, if someone viewed 10+ problems without being able to answer any of them, we believe it would be better to work on the fundamentals first. The system that you mentioned was abused by members who unable to answer any of the questions, but artificially maintained their level due to not attempting sufficient number of them wrongly each week. Calvin Lin Staff · 3 years, 9 months ago

Log in to reply

@Calvin Lin As I have already mentioned somewhere else, I think if users are allowed to level down own their own (but not level up of course!) then this issue will be solved quite easily. Why not to do that? Snehal Shekatkar · 3 years, 9 months ago

Log in to reply

@Snehal Shekatkar With the current ratings system, it is clear that anyone can easily (manually) level down by viewing enough problems / getting them wrong (as opposed to just stating it in the FAQ). They do not need to wait 2 or more weeks. Calvin Lin Staff · 3 years, 9 months ago

Log in to reply

@Calvin Lin I think users should have the option to manually level down if they want to, instead of arranging it by getting hard questions wrong on purpose. Matt McNabb · 3 years, 9 months ago

Log in to reply

@Calvin Lin Hmm.. I have one more question.. if I view the same problem more than once.. let us say I close my browser and reopen, then will my rating reduce every time? Snehal Shekatkar · 3 years, 9 months ago

Log in to reply

@Snehal Shekatkar Nope. It only decreases once (during the initial view). Silas Hundt Staff · 3 years, 9 months ago

Log in to reply

@Snehal Shekatkar I would like that! Wei Jie Tan · 3 years, 9 months ago

Log in to reply

@Calvin Lin That's fair enough for me. I was expecting a much more stringent rule.

But, I request you to create a page listing all such rules and guidelines of the site and updating them whenever necessary. Sambit Senapati · 3 years, 9 months ago

Log in to reply

A suggestion: shouldn't our rating increase when we publish a solution to a problem? Publishing, or even writing out a solution verifies that you were able to solve the problem without relying on things like computers for symbolic manipulation or anything of that sort, and are able to communicate your method clearly. (Yes, I know there are a couple of solutions out there that list programs to solve a problem, but those are in a very small minority.) Tanishq Aggarwal · 3 years, 9 months ago

Log in to reply

@Tanishq Aggarwal The problem is that then a lot of people would post poorly-written and erroneous solutions. Alternatively, upvotes could increase rating. Daniel Chiu · 3 years, 9 months ago

Log in to reply

@Daniel Chiu That works too. Tanishq Aggarwal · 3 years, 9 months ago

Log in to reply

Its really amazing! I dont know in which game it is used but I know that brilliant.org would be the first site to do this. HATS OFF TO BRILLIANT!!!!!! Anshul Agarwal · 3 years, 9 months ago

Log in to reply

Great improvement. I was wondering, does my rating increase more if I solve a problem with a higher rating? I suppose yes, and if so, does my rating increase even more if a problem turns out to be harder than at the time I solved it? Tim Vermeulen · 3 years, 9 months ago

Log in to reply

@Tim Vermeulen Yes. Solving a higher rated problem increase your rating by a larger extent, as stated in "what's this".

No. Your rating is only affected at the point when you (initially) viewed or answered the problem (both correctly and incorrectly). Viewing the question repeatedly (whether you solved it or not) will not affect your rating. Calvin Lin Staff · 3 years, 9 months ago

Log in to reply

Comment deleted Nov 03, 2013

Log in to reply

@Tim Vermeulen Oh sorry, I misinterpreted your question. Your rating doesn't change in the future, from actions in the past.

Let me update my reply Calvin Lin Staff · 3 years, 9 months ago

Log in to reply

@Calvin Lin Okay! Yeah, that makes more sense. Though, I have noticed some problems having a very high rating just at the start, when nobody's solved it yet. That would mean that solving problems very early during the week could result in a rapid rating increase? Tim Vermeulen · 3 years, 9 months ago

Log in to reply

@Tim Vermeulen There are (will always be) edge cases in any such system. Several factors are involved, and we try and account for them.

Problem ratings at the start of the week are extremely volatile, since we have merely set an estimate, and are letting it stabilize to it's own rating. If we are uncertain about the rating of a problem, then it would have a smaller effect on your rating. Calvin Lin Staff · 3 years, 9 months ago

Log in to reply

@Calvin Lin You could calculate people's rating change after a few days, when the problems rating is settled, or even at the end of the week.

That way everyone will be treated equally when dealing with rating changes. Ton De Moree · 3 years, 9 months ago

Log in to reply

When is your level updated to match your rating? From my ratings, I should be a Level 4 in Physics and a Level 3 in Combinatorics, but instead I'm at Levels 3 and 2 respectively.

And why do the problem ratings typically fall outside the band for the level? For example, this week's Level 3 Algebra problem ratings are (at the time of writing) 1934 (Level 4), 1696 (Level 3), 2151 (Level 5!) and 2225 (also Level 5!). Only one of them falls within the band for Level 3! John Khoo · 3 years, 9 months ago

Log in to reply

@John Khoo Hi John,

When is your level updated to match your rating? From my ratings, I should be a Level 4 in Physics and a Level 3 in Combinatorics, but instead I'm at Levels 3 and 2 respectively.

Sounds like a bug! We'll check it out, thanks for writing.

And why do the problem ratings typically fall outside the band for the level? For example, this week's Level 3 Algebra problem ratings are (at the time of writing) 1934 (Level 4), 1696 (Level 3), 2151 (Level 5!) and 2225 (also Level 5!). Only one of them falls within the band for Level 3!

We are still tuning the ratings system. Note that a problem's rating is not necessarily supposed to line up with the user levels.

The important question for you is: do those problems feel too hard for you? Silas Hundt Staff · 3 years, 9 months ago

Log in to reply

@Silas Hundt I found another BUG here which I exploited once ...

In a physics problem I had used up 2 tries guessing the answers as A and B and I knew that my answer could be either of X or Y but because of only one try left I thought of the following...

I opened the same problem page in 2 tabs and gave the answer as X in one and as Y in the other and pressed enter in both almost at the same time .. Both pages showed that all my answers were wrong but also something else one showed that I had entered A,B & X the other showed that I had entered A,B & Y ... So I actually had used 4 tries and couldn't get to the answer.. I hope you fix this ;-) Santanu Banerjee · 3 years, 9 months ago

Log in to reply

@Santanu Banerjee The "Bug" is that the server can handle more than one request at the same time. While this may make it seem like you get 2 tries, whichever came in slightly after the other (from the perspective of the servers) would end up over-writing the other so it would basically be like tossing a coin between the two.

It is always a better idea to enter just the one answer that you think is more likely to be correct. Sam Solomon Staff · 3 years, 9 months ago

Log in to reply

@Silas Hundt Hmm...all my levels were upgraded accordingly when the new week started, except Physics. Rating-wise, I should be Level 4, but instead I am still Level 3. John Khoo · 3 years, 9 months ago

Log in to reply

@John Khoo According to your profile, you are now Level 4 in Physics! Silas Hundt Staff · 3 years, 9 months ago

Log in to reply

@Silas Hundt But the Challenge page still says I'm Level 3, and I got the Level 3 problems this week... John Khoo · 3 years, 9 months ago

Log in to reply

@John Khoo Yep. Your problems for the week are decided by your level at the beginning of the week. Next week, you will receive the Level 4 problem set. Silas Hundt Staff · 3 years, 9 months ago

Log in to reply

@Silas Hundt Ah, I see. Thanks for clarifying. John Khoo · 3 years, 9 months ago

Log in to reply

@Silas Hundt Well, they were OK for me. But my main problem was really with the disparity between rating and level. But since you lot will be tuning the ratings, I'll just wait and see what happens. Thanks for responding. John Khoo · 3 years, 9 months ago

Log in to reply

@John Khoo I don't think there is a level 6 or 120... Ryan Wood · 3 years, 9 months ago

Log in to reply

How can I know my rating? Zi Song Yeoh · 3 years, 9 months ago

Log in to reply

@Zi Song Yeoh Right now, the place to see your current rating in a topic is next to any weekly problem. Silas Hundt Staff · 3 years, 9 months ago

Log in to reply

@Silas Hundt What about in your profile next to your level. Ryan Wood · 3 years, 9 months ago

Log in to reply

@Ryan Wood Soon! Silas Hundt Staff · 3 years, 9 months ago

Log in to reply

I think you could add Selling Tips because when people do not know where to start, Only with Shopping Tip the Person has a Greater Sense about What Problem Want ! Gabriel Merces · 3 years, 9 months ago

Log in to reply

I've noticed that over the past few days my physics rating has been fluctuating even though I haven't looked at or attempted a problem since the beginning of the week. The last time I finished with physics for the week, my rating was slightly above 1500, about 1503. Then by yesterday it had gone down to 1497 and today it was gone back up to 1499. Why would it be changing like this? Cole Coupland · 3 years, 9 months ago

Log in to reply

Why is that the ratings of problems change so fast ? Mirza Baig · 3 years, 9 months ago

Log in to reply

If we have the rating 1750-2000, doesn't it imply that we are in the 5th level? Sambit Senapati · 3 years, 9 months ago

Log in to reply

@Sambit Senapati We have shown your rating on the challenges page, along with the next level up target.

If your rating is in the range 1750-2000, then you would be in level 4. There is a slight allowance for level 5 users who had a small drop in rating, due to getting a problem wrong or viewing more problems. So it's possible to have a rating just under 2000, while still be in level 5.

If you look at the levels page, we stated level up targets, which are nicely rounded off to 1250, 1500, 1750, 2000. Calvin Lin Staff · 3 years, 9 months ago

Log in to reply

@Calvin Lin Yesterday, it showed "level up at 1750" when I was in level 4 in number theory. Hence my question.

It is resolved now. Sambit Senapati · 3 years, 9 months ago

Log in to reply

I have a complaint. That is every time the server updates, my rating goes down. And it has happened twice now. First I had 1731 for number theory, then went down to 1681 and now 1641. It seems it won't be long before i level down to level two. This has happened with all the other topics too. I was meant to be on level three number theory, algebra and number theory. It will be good if you could help me with it. Sabab Ahad · 3 years, 9 months ago

Log in to reply

@Sabab Ahad If you read the "What's this" explanation, your rating goes down when you initially view the problem, or answer it wrongly. It goes up when you answer the problem correctly. I've reviewed your rating changes, and they are occurring as intended by our algorithm. Calvin Lin Staff · 3 years, 9 months ago

Log in to reply

@Calvin Lin Ok, I understand, thanks for your help. Sabab Ahad · 3 years, 9 months ago

Log in to reply

Can I get degrade immediately in Number Theory? Patrick Engelmann · 3 years, 9 months ago

Log in to reply

My Algebra has just been upgraded to Level 5. However, my rating is [removed - Calvin]

Shouldn't my Algebra be at Level 4? I mean, I like having my algebra rated highly, and personally I think I'm good at it (my favourite), but Algebra Level 5 is 2000 and above, right?

Just pointing out a possible error in the system Eng Ngee H'ng · 3 years, 9 months ago

Log in to reply

@Eng Ngee H'ng I see that your rating did cross over 2000 when you solved a probelm, then came down slightly when you viewed a new problem.

There is slight leeway in lowering a level, just so that you do not aggressively toggle back and forth between levels when you are right at the threshold. Calvin Lin Staff · 3 years, 9 months ago

Log in to reply

@Calvin Lin There is another issue with the "viewing penalty" namely viewing problems that are in other problem sets, like a level 5 combinatorics set for instances. When I was looking at a level 5 Number Theory problem the systems took off rating points. Nikhil Pandya · 3 years, 9 months ago

Log in to reply

@Nikhil Pandya How can you see level 5 problem if you are on level 4 (or that matter any lower level) ? Mirza Baig · 3 years, 9 months ago

Log in to reply

@Mirza Baig Go on the profile's of those in the respective level you want to observe, check their activity and you'll see the problems they have solved, e.g. the ones in that level. Nikhil Pandya · 3 years, 9 months ago

Log in to reply

A question and clarification. Do the ratings fall when you do not attempt any question? As it turns out that i had scaled the barrier from one level to another but when the new week came my ratings had fallen. Sourav Chaudhuri · 3 years, 9 months ago

Log in to reply

@Sourav Chaudhuri If you view the problem i.e. open it's link, your rating would fall, even if you didn't try to answer. If you get it wrong, your rating would fall even more. If you get it right your ratings would increase significantly enough to compensate for both reductions.

If you do not open problems link and thus don't see the problem, your ratings would not be affected. Mirza Baig · 3 years, 9 months ago

Log in to reply

Just had an idea: what if, instead of having problems by level, you could have problems by rating? I mean this in the sense that you wouldn't get "Level 4 algebra problems" but "Rating 1600-2100" problems, or something like that. Matthew Lipman · 3 years, 9 months ago

Log in to reply

The rating system is simply awesome! Nice work brilliant team. A question i have in my mind- how much my rating increase whenever I solve a problem? Dhruv Bhasin · 3 years, 9 months ago

Log in to reply

Love the new rating system. I think it is great!

This question is a bit off topic, but I was wondering why there are no more geometry questions like there used to be? And are they going to be brought back? Nico Stirling · 3 years, 9 months ago

Log in to reply

@Nico Stirling Also I can't access the Data Structures and Algorithms questions. Is this the same for everyone or is it a bug? Nico Stirling · 3 years, 9 months ago

Log in to reply

@Nico Stirling Same for everybody.. it will come back soon.. Snehal Shekatkar · 3 years, 9 months ago

Log in to reply

@Snehal Shekatkar Ok. Thank you :) Nico Stirling · 3 years, 9 months ago

Log in to reply

@Nico Stirling Why have I gotten so many vote downs? :( Nico Stirling · 3 years, 9 months ago

Log in to reply

@Nico Stirling Probably because your question was not on topic, and they have been asked many times before. Tim Vermeulen · 3 years, 9 months ago

Log in to reply

Hi. I just saw some comments about the "views" and I got an idea, so I want to share it. Maybe, if the problems are attached in a PDF document, the users will get able to see them without starting the weekly challenge. That could be great, because then the rating will be evaluated correctly. If a user opens the PDF just because he/she wants to see the problems, he or she will not be penalized. But if he/she starts the challenge, it will mean that he/she knows some of the problems, so in case of missing them, the user can be penalized justly. How does it looks? Diego E. Nazario Ojeda · 3 years, 9 months ago

Log in to reply

@Diego E. Nazario Ojeda What's the difference between this and just taking away the viewing penalty? Matthew Lipman · 3 years, 9 months ago

Log in to reply

@Matthew Lipman Well.. you'll be able to decide if whether you are prepared or not. I think the penalty is justified after you miss a problem, and it is possible only if you see it in other way, different than starting the challenge. It's just an idea... Diego E. Nazario Ojeda · 3 years, 9 months ago

Log in to reply

@Diego E. Nazario Ojeda If one is not prepared one should be penalized and later sent in down level. You can see problems one by one while solving each before viewing the next. That way you only lose for actually not doing the problems instead of just not having time.

Anyhow Brilliant has a pressure of maintaining sufficient traffic of students actually solving problems, so this makes lot of sense from their perspective. Mirza Baig · 3 years, 9 months ago

Log in to reply

@Mirza Baig With prepared i meant like being prepared to start working with the problem on that specific moment. Diego E. Nazario Ojeda · 3 years, 9 months ago

Log in to reply

@Diego E. Nazario Ojeda Well you have entire week :) Your ratings would be deducted only once for viewing. You may few it thousand times (or more) after that without any loss. Mirza Baig · 3 years, 9 months ago

Log in to reply

Due to this new ratings system, I've been leveled down to level 4 in algebra. Now, I'm pretty darn good at algebra. In fact, I just finished all 4 algebra level 4 problems this week in ≤ 15 minutes. Please level me back up to level 5 in algebra. Cody Johnson · 3 years, 9 months ago

Log in to reply

@Cody Johnson I see that you are now level 5 in Algebra. If you believe that this is still an issue, send me an email.

Note that your ratings will move around some during the week, as you work to solve problems. Calvin Lin Staff · 3 years, 9 months ago

Log in to reply

×

Problem Loading...

Note Loading...

Set Loading...