Waste less time on Facebook — follow Brilliant.
×

Brilliant Ratings

I think the Ratings on brilliant go down too much when you get a problem wrong and doesn't go up enough when you get a problem correct. Sometimes when I get a problem wrong the first time and then get it correct the second time I end up with less rating than i started with

Note by Yan Yau Cheng
3 years, 8 months ago

No vote yet
1 vote

Comments

Sort by:

Top Newest

Yes! Thank you for voicing something that almost everyone else, I think, has been wondering about. I contacted Silas, and he claims that you never lose ratings for a correct answer, even on the the third try, but nonetheless it only goes up by 1 or 2 points. It's obnoxious because it doesn't encourage people to try new things, and experiment with a new area of math because they don't want to lose an obscene amount of points.

Finn Hulse - 3 years, 8 months ago

Log in to reply

There is the possibility that this is a front-end issue, where we do not update your displayed rating aggressively, especially for old pages / problems. and I'd have to look further into that. In your specific case, see other reply below.

For 'older' users who have done 5 problems in the topic, I do not see any evidence in the backend for them to lose 50+ points on a problem, unless their rating is significantly higher than the problem.
For new members, their rating change is more aggressive as we do not have much information about them as yet, other than the diagnostic problem which they did.

For 'older' users, if you answered the problem correctly in your second time, your net rating change would be positive. If you answered the problem correctly in your third time, your net rating change should be negligible (well within \( \pm 5 \) ).
For new members, this may not be the case, as the aggressive rating changes could penalize you at first, before you finally get the problem correct. However, this edge case is extremely rare.

Calvin Lin Staff - 3 years, 8 months ago

Log in to reply

And when I get 3 wrong on one problem, that can be as much as 40-100 points off! And I get all 3 wrong a lot!

Finn Hulse - 3 years, 7 months ago

Log in to reply

I think we should redo the whole determining your level thing for each topic because the point system is kind of screwed up.

Victor Song - 3 years, 8 months ago

Log in to reply

Agree

Anirudha Nayak - 3 years, 8 months ago

Log in to reply

Your rating should never decrease upon solving a problem correctly on the second try (unless the rating of the problem has drastically changed). The next time it happens, can you send me an email listing the problem that you worked on? If your rating in the backend went up, then this could arise from information not updating aggressively enough (or a display issue). Try refreshing the page and see if your rating changes.

For (long term) users whose ratings have stabilized, then the reward/loss of ratings indicate the relative chance that you will be getting the problem correct. As such, for a person who works only on very easy (relative to his level) problems where he "should be" getting them right 90% of the time, he might see a -45 if wrong and a +5 if correct, which leads to a net of 0 on expectation. Conversely, if you wok on a problem which you should only be getting right 10% of the time, then you should see something like +45 if right and -5 if wrong. (Rating change values not necessarily accurate)

Calvin Lin Staff - 3 years, 8 months ago

Log in to reply

Actually, I have lost 50-200 points for a wrong answer. It's not fair. I mean, people don't actually get stupider. How can you go and try new things, and try your hand at something you're not familiar with if you're afraid of losing huge numbers of points. Please have Silas forward my emails that I sent him to you discussing this problem, and my solution. Thanks!

Finn Hulse - 3 years, 8 months ago

Log in to reply

I have looked into your ratings of Combinatorics in the past 30 days, and at no point in time did your rating decrease by 50 or more in the past month. You do have a significant amount of wrong attempts, though there are very few in which you lost 30-50 points. Even in those situations, the problem was extremely easy and when you next got it correct you gained the points back.

Over the period of Jan 30 to Feb 2 (midday), your rating dropped from 1500 to 1200. During that period, you viewed 48 problems, had 21 eventually correct, and submitted 62 wrong attempts (3 wrong attempts to several problems). The significant number of wrong attempts was responsible for your rating drop. It did not happen during 1 problem, but over a period of time. As mentioned, it could appear to happen after 1 problem due to your displayed rating not updating.

Looking further into it, on Feb 2, over the span of 30 minutes, you had viewed 60+ problems in Combinatorics and given 3 incorrect answers to each of them. This resulted in your rating dropping by 1000 points in total. (You initially lost 30-50 points per problem from 3 wrong attempts, and that tapered off as your rating dropped very low.)
I see that we did a manual reset to your Combinatorics rating after that. Since then, I see 6 correct answers (each of which gave you 30-50 rating) and 60+ wrong attempts. Over this entire period, your net rating change is negligible. However, you lost up to 100 rating, before regaining it back towards the end.
If this is not how you have used the site, you should change your password / check if someone else had access to your account.

A possible explanation as to why you see the rating drop in 1 question, is that we might not be aggressively updating your rating changes as you work on different problems. For example, if you open Combinatorics problem #1 with a rating of 2000, and then get 20 problems wrong so that your rating is now 1800. When you answer Combinatorics problem #1 wrongly, it would appear that your rating dropped from 2000 to 1800. Given your pattern of answering, I believe that this is the most reasonable explanation.
We do not make ratings update constantly as that will be a big strain on our servers. However, we could make ratings update upon refreshing of the page, and ensuring that the cached value is not too old. I will look further into this.

Note: initial rating changes when you first joined are much more aggressive, since we only had a diagnostic problem to base it on. At that stage, answering a simple problem wrongly can change your score by 50-100, which allows us to better sort people into the correct ability group.

Calvin Lin Staff - 3 years, 8 months ago

Log in to reply

@Calvin Lin See my comment below, please.

Finn Hulse - 3 years, 7 months ago

Log in to reply

@Calvin Lin About the 1,000 rating loss... I was proving a point to Silas that just because somebody gets a couple of problems wrong doesn't mean that they're any dumber. Which is completely true. Nobody really does get any dumber, (excepting NFL players), so when I lose points for trying things that I haven't had a lot of experience with, I get frustrated. I really don't like losing points at all, and I am the type of person who likes to try new things, so I think it's really bad that Brilliant, a site for broadening math abilities, apparently doesn't want users to try things they aren't great at. What I suggested to Silas is that instead of tweaking the ratings system to lose less points, that the whole entire system is revamped, and the diagnostic test consists of 10 or so questions, and from whatever rating you get on that, you can only improve, but usually no more than 5 points, and you only get points for problems that are higher than your rating. I think this would be great, because it would be a more accurate diagnostic test (not just 1 problem), and you could try new things without worrying about ratings.

Finn Hulse - 3 years, 8 months ago

Log in to reply

I partially agree with this, but you need to be careful because (in theory) a person who is supposed to be Level 3 could be able to "farm" Level 1-3 problems to reach Level 5.

Trevor B. - 3 years, 8 months ago

Log in to reply

Hi everybody,

Thanks for pointing this out (again) and being patient.

We constantly making adjustments in the background and tuning the rating system. There will be bigger changes this coming week. As Trevor and others pointed out, there isn't a "simple" fix to ratings—it is a dynamic system, so it needs to be adjusted iteratively.

Silas Hundt Staff - 3 years, 8 months ago

Log in to reply

\[\int_{i}^{agree} {withyou}^{2} \]

Ruhan Habib - 3 years, 8 months ago

Log in to reply

I'm used to it because we can have another chance but for multiple choices it's crazy!

Christopher Boo - 3 years, 8 months ago

Log in to reply

I agree. I really don't like the format of Multiple choice problems. These type of problems should appear only on an entrance exam.

People just seem to guess the answer sometimes. Upon that, we just have ONE chance!

Anish Puthuraya - 3 years, 8 months ago

Log in to reply

I agree... 4 chances will be more than enough.

Satvik Golechha - 3 years, 7 months ago

Log in to reply

i totally agree.... if they have to post Multiple choice there shouldn't be any points deducted or given for the problem

Kartik Tyagi - 3 years, 8 months ago

Log in to reply

It doesn't make much sense some of the time, because you kind of get scared to answer a problem. I personally don't have my score go to far down all of the time, but the amount that I've heard people lose on this post seems insane. Also, when somebody loses enough points to level down, it says that they have leveled up.

Robert Fritz - 3 years, 7 months ago

Log in to reply

Really?

Finn Hulse - 3 years, 7 months ago

Log in to reply

Well i request all of you to check out those Mechanics level-I questions. Those questions are like more of your guesses than physics but if you get 1 right, your rating goes up by a point(many times it doesn't) but if you get one wrong, atleast 30 points are deducted.

Saqib Mohammed - 3 years, 7 months ago

Log in to reply

Yeah, I hate that.

Finn Hulse - 3 years, 7 months ago

Log in to reply

I also agree with this.

Lee Wall - 3 years, 8 months ago

Log in to reply

totally agree with u. i had rating 1850, then i got a 2380 problem wrong and then my rating went1300

Omkar Pande - 3 years, 8 months ago

Log in to reply

The situation that you described should never be possible. Can you give me the question in which you saw this happen?

Even if a person with rating 3000 got a problem with rating 1000 wrong, I would be surprised if his rating dropped by more than 150.

A slight possibility is that the topic of the problem was updated (but this happens very rarely). For example, if the initial topic was Number Theory (where you are level 4), and it gets updated to Computer Science (where you are level 1), then you might see your displayed rating change from 1800 to 1200.

Calvin Lin Staff - 3 years, 8 months ago

Log in to reply

I'm not actually remembering the problem. But as far I do remember, I had put the answer in fractions (in the form a/b) instead of decimals. Can this result in a drastic change in the rating?

Omkar Pande - 3 years, 8 months ago

Log in to reply

@Omkar Pande I have looked into your ratings. At no point in time do I see that answering a problem decreased your rating in the topic by more than 100. (Not that as you are a new member, rating changes can large initially, like 60-100, to get you to a suitable level, after which it tapers off and should be closer to 30)

I believe that if you tried entering 3/5 (instead of 0.6) our system would not accept it. We only allow for decimal / integer answers. Can you clarify what you mean?

No. All wrong answers are treated equally. It doesn't matter whether you made a calculation error, a logical error, or a typo.

Calvin Lin Staff - 3 years, 8 months ago

Log in to reply

@Calvin Lin I don't know man! You are saying it so surely that now I am starting to doubt my own memory. I think I'll just let it be and try to gain back my rating. Sorry for any inconvenience.

Omkar Pande - 3 years, 8 months ago

Log in to reply

Wrong. I lost 300 points in my combinatorics. Silas put me at 1500, and it dropped to 1200 after one problem.

Finn Hulse - 3 years, 8 months ago

Log in to reply

@Finn Hulse And is that 3000 level person actually 150 points stupider because of one problem? People don't get stupider (except for NFL players). People only get smarter. Ask Silas about those emails!

Finn Hulse - 3 years, 8 months ago

Log in to reply

@Finn Hulse Edit: I consolidated my response above.

Calvin Lin Staff - 3 years, 8 months ago

Log in to reply

I partially agree with the above statement; however, I think everyone needs to realize how much Brilliant has tried to resolve this issue in the past several months (it was much worse when I joined Brilliant in October). Basically, what I'm saying is give it a little bit of time, and if nothing has happened, then you should bring this issue up again.

Sam Thompson - 3 years, 8 months ago

Log in to reply

I agree with you. Thanks for mentioning it.

Tahsin Ahmed - 3 years, 8 months ago

Log in to reply

Agree!!

Shabarish Ch - 3 years, 8 months ago

Log in to reply

ya wright bro

Abhijeet Singh - 3 years, 8 months ago

Log in to reply

I would like to type here that brilliant is not giving prizes now i suggest brilliant to be back with there prizes

Anubhav Singh - 3 years, 8 months ago

Log in to reply

×

Problem Loading...

Note Loading...

Set Loading...