Yes, sometimes you mean something literally–but most of the time you don’t.
According to Vocabulary.com:
Figuratively means metaphorically, and literally describes something that actually happened. If you say that a guitar solo literally blew your head off, your head should not be attached to your body.
This Atlantic article illustrates how we use words as crutches: How many times a day do you say like, um, uh, yeah, and literally without being aware of it? And before you delude yourself that actually is any better, think again:
Actually is a word worse than literally because it is a secret criticism, an indirect jab, a correction with a barb. … “Actually, I’d rather have pizza,” can be said just as easily as “I’d rather have pizza,” so why say actually at all? If you really find yourself in a situation in which you need to say actually, how about choosing, simply, really or truly, or perhaps verily or even forsooth instead.
OK, I don’t think I’ll be working forsooth into everyday conversations, but that writer just may have been throwing it out there facetiously.
So, does that mean it’s wrong to use literally or actually in everyday parlance?
I’ve literally never tasted anything so good!
I literally think I’m going to starve to death if our food doesn’t come soon.
You didn’t know Saditha was pregnant? Literally?
Though the above sentences are a little painful to read, my answer is no. Go ahead and use these words as long as you know you’re using them wrong. But the next time you get the urge to say, “I literally have nothing to wear,” just keep in mind that, “Figuratively speaking, I have nothing to wear,” is actually what you should have said.