Picking off the aphids seemed to work.

It was a bit of work but I think I got them before they infested the tomato plants.