To the extent that he did mean to indirectly correct you, he is merely expressing a cultural/socioeconomic preference.
I do too.
I do also.
I do as well.
I would guess that most speakers would probably rate these three options in this order on a scale that ranges from "common/pervasive/typical" to "more refined."
All are perfectly correct. All are perfectly common.
I do not perceive any sort of widespread notion that "too" is prime marker for "low" speech. "As well" is simply perceived as more sophisticated (and by so being perceived that way, it is).
In other words, even people who find it a helpful thing to explicitly 'prefer' "as well" would (very likely) still use "too" themselves in various (perhaps unguarded?) situations.
This is not a big deal. But when you do find yourself in the company of well-heeled acquaintances, using "as well" more often might very well bump their intuitive judgement of your diction up ever so slightly. Though, among the common folk, too much of it might do the opposite.
The crowd never desires less than complete conformity, and no individual ever completely conforms, and crowds change their minds by the day.