The reason he looked like he had no idea why you were scolding him was because it happened 'after' the incident. Unless you catch him in the act you cant scold him and if you do catch him it must be an immediate response from you.
My only suggestion would be only allowing him outside under your supervision, that way you can keep an eye on him and tell him 'NO' if he goes near it or attempts to undo what you've done.
With regard to the plants, i agree with Anne, fence them off so he cant touch them.
He sounds like a lovely boy who enjoys spending his time with you
. I agree with you on keeping him outside when you are outside, just make sure you either keep a very close eye on him, leash him or put him in a run. That way he can still be with you.
Good luck! I hope you get some other advice that can help you.
Trina