I get them any time I have outdoor plants. Might not be the same type of caterpillar but sure sounds like what I deal with.
My remedy is hunt them down every day, pick them off, and step on or squish them. They'll always be in the upper parts of the plants, the tallest growing tips, and can cocoon by wrapping themselves in a leaf blade with their webbing.
BT caterpillar killer can work to kill and repel and their natural predator are wasps.
If they're like the worms I get you must get them off when you find them. They dont like cannabis resin and will burrow down inside a bud to eat the stalk and kill the bud which kills the resin. The result from the burrow and chewing will look like bud rot or mold and cause dead loss in the harvest.
More than likely they came from a moth(s) laying eggs on your plants. Make sure moths cannot get inside your grow space and if outside a bug zapper can often help get the moths before they lay eggs but still gotta search them out every day.