×

# A problem for Level 5 users in solving low Level problems

Does it seem fair that if you're at Level 5 in some category, and you answer "wrongly" some problem in Level 1, your rating can drop as much as 34 points, and yet if you had answered "correctly", your rating won't go up at all? Doesn't that strongly discourage anyone from trying any problems at all that are below their Level?

A recent algebra problem posing a simple linear equation has the so-called "correct" answer of x = 0, when in fact any value of x would be a solution. An Algebra Level 5 user can lose as much as 34 points in tangling with this bogus question.

In game theory, one looks at expected returns versus expected potential losses. Since low Level problems return nothing in rating at risk of a considerable drop in rating, as compared to higher Level problems, doesn't that mean that the best strategy is to avoid the lowest Level problems whenever possible? Especially when a significant percentage of them are poorly worded or even have wrong "correct" answers?

I would suggest that deductions in ratings for wrong answers should be the same or about the same as increases in ratings for right answers.

Note by Michael Mendrin
2 years, 12 months ago

Sort by:

I agree, I think the rating system is way too harsh. I find it's difficult to maintain my level because even though I may answer 3 level 5 problems correctly, getting one level 3 problem wrong seems to negate all of the gains that I made, even if it was due to a simple reading error. I think the rating system needs to be modified so that either it is less harsh on getting lower level problems wrong or more generous when you get higher level problems right. The latter would probably be more encouraging for users of the website but maybe a bit of both would work. · 2 years, 12 months ago

If all the problems were perfect and thoroughly vetted, as if taken from Olympiad problems, then it could be tolerable, because then at least I can trust to know exactly what the problem is, and that it has an unique, correct answer. Unfortunately, that's far from the case in Brilliant, where even a lot of the rated problems are defective and you just don't know you're getting into to. We shouldn't be punished for having tackled defective problems. No, the Brilliant staff doesn't have the time to address each and every complaint about such bad problems, there has to be a better way. One way would be to make the rating system a little more realistic and less harsh. · 2 years, 12 months ago

That's what I meant by my comment. · 2 years, 12 months ago

Well, I'm trying to completely concur, Cole. By the way, thanks for the interesting problems you've been posting. They've been fun. · 2 years, 12 months ago

I'm glad you made this post, this issue has been on my mind for a while. It was good to see that you made a solution to my center of mass question, you got my upvote. · 2 years, 12 months ago

Brilliant has updated the algorithm it seems, all thanks to @Michael Mendrin ! and all who reshared and liked the note.. · 2 years, 11 months ago

hey there I am Salman Zafar I am new here can you just help me out ? I will be verrrrrrrry thankful · 2 years, 10 months ago

hey there I am Salman Zafar I am new here can you just help me out ? I will be verrrrrrrry thankful · 2 years, 10 months ago

Thanks for your feedback.

1. The system is set up such that the expected gain from solving a problem is 0. For example, if we're 99% sure that a level 5 can solve a level 1 problem, then if they get -34 for a wrong answer, they should get +0.345 for a right answer. While we round your rating off when displaying, fractional changes are still recorded.

2. I agree that bogus problems are a concern. Your rating will update when the answer is corrected. This often rewards you, due to the fake high rating of the problem.

3. We are also working on providing more clarity on the validity of problems, and will provide more tools to the community to moderate problems.

Staff · 2 years, 12 months ago

This policy of "0 expected gain" makes sense only if all of the problems are perfect (i.e. clearly and unambiguously worded, with correct answers). But since we know that a good percentage of them are not, this needs to be factored in. Let's imagine that I'm at some casino playing the roulette table, the odds of, say, 00 winning is 1/37, but the payout is 35 to 1, slightly less than 0 expected gain (because the casino has to make money). It's a risk I'm willing to take, because then I can expect to play for a while and have fun before losing all of my money. However, if the odds of my winning is, say, 1/40 (which is about what happens when I tackle Level 1 problems, because of problems with poor wording or incorrect answers), then there's going to be a lot less than 0 expected gain, and I'm going to find out that my money is going to run out in a hurry. And so I'll leave for a more fair casino.

The Brilliant staff has resources to understand and use statistics, and my advice would be that Brilliant uses the same quality control philosophies that manufacturers have, which is that they don't start out assuming that they are delivering perfect products every time. They have to factor in the reality that a certain percentage of their products are doing to have defects, otherwise they are going to lose customers. If I were on the Brilliant staff, I'd be focused on making my customers happy, instead of ignoring what it is like for consumers to use Brilliant. I am one consumer, and I would have to say this is probably the most aggravating characteristic of Brilliant, which is that I cannot just go and tackle problems "for fun", because I know how dangerous it can be if I happen to run into a bogus problem. Stepping on a land mine once in a while is okay, but after it's blown up in your face too many times, well, you start avoiding the minefields, and look elsewhere. You might already be losing consumers after they've tried out Brilliant for a while.

My strong suggestion to the Brilliant staff is that they pay attention to what makes their consumers happy, not what "ideals should be upheld". There are ways to do both. Be creative. · 2 years, 12 months ago

How about marking a user posted problems after they passed a quality check with some icon indicating that the answer is correct and that the question contains no ambiguities. · 1 year, 10 months ago

I think the multiple choice problems should all be unrated. I remember going down like 300-400 in calculus because I got a MC problem wrong. · 2 years, 11 months ago

I think that questions that answer as explanation should be a discussion instead, not a multiple choice. People with most votes gets a ratings and stuffs. I made a very stupid mistakes on easy problems and the score dropped down like 250 by just a single problem. Honestly, I remembered of having a nightmare after that problem on that night lol. · 2 years, 7 months ago

Haha but you are back at 5. · 2 years, 10 months ago

Yup I agree. I also experienced this drastic rating decline in algebra when I by mistake clicked on the wrong option of an MCQ while attempting a level 2 ques. (I was level 4 on algebra then..) Unfortunately fell to Level 3 now... · 2 years, 12 months ago

hey Vishal I am Salman Zafar I am new here can you just help me out ? I will be verrrrrrrry thankful · 2 years, 10 months ago

I know its really a very big problem these days, that's why I never even attempt level 4(or below) questions these days(only 5's) as they're much more profitable. · 2 years, 2 months ago

You won't believe me, but this very feature of Brilliant has cured me of my really bad habit of answering the questions whose answers are doubtful to me. It helped me greatly in the exams which have negative markings. Some of my great downfalls are -

Combinatorics - Level 3 to Level 1

Mechanics - Level 4 to Level 3

Electricity & Magnetism - Level 2 - Level 1

How I am thankful to Brilliant, I can't tell you. (I am not being in the least sarcastic here, seriously!). · 2 years, 11 months ago

Can't say anything, but I agree with this. · 2 years, 11 months ago

Indeed, it seems like there is more penalty for incorrectly answering a problem than there is reward for correct responses. In my opinion, the best solution to this would be to increase the number of points gained by answering harder problems and change the system so that points are not lost by answering incorrectly until all three tries have been used. · 2 years, 11 months ago

Excellent idea Lee! @Peter Taylor have you ever considered this? · 2 years, 10 months ago

Seriously. I agree SO much. · 2 years, 12 months ago

Same here · 2 years, 10 months ago