Weight loss is a factor of both calories in and calories out. You've been working on the second half of the equation and that's great, but neglecting the first half isn't a good plan in the long run.
Something to think about, though, is that muscle is more dense than fat. You may not have been seeing any movement on the scale because you've been building up muscle mass. As you noticed that your clothes are fitting better, it is very likely that you've been adding muscle. More good news is that muscle uses more calories at rest than fat does.
Don't give up on the exercise completely! You still need the second half of the equation. Still, a couple of rest days a week is a very good idea so that your muscles can recover and build themselves stronger.
You're not doing badly at all! It takes a while to make the changes necessary for a healthy lifestyle and you're making those changes.