I doubt you will find any solvent that's going to wash latex off without being strong enough to attack what's under it. Alcohol and stuff like Goo Gone may make the surface tacky, but that's probably about all, and you would be there for decades rubbing it. The problem is that in order to wash off dried paint, you have to replace the solvent, moisture or whatever was originally in it to make it soft or liquid. Dried paint doesn't suck up solvents anywhere near fast enough to do this. It takes strong chemicals, like those in stripper, which really attack and liquify the old finish - and will most likely also go after your filler.
I don't think there is much you can do other than abrade it off (sand, scrape or whatever else you've got). I suspect that a good cabinet scraper may be the fastest, most controllable tool to get the latex off without tearing up what's under it.