Do colleges actually focus on the difficulty of your courses?

It would not look good to college to see your grades have dropped as you are a junior and senior. You are older, have more education and should be able to handle the junior and senior level cles. College level cles are harder, are you grades going to keep going down because the cles get more rigorous?