When Locke makes his inference to materialism as the best explanation, the inference is one about causation. The materialist inference that he thinks the evidence warrants is this: the reason that sensory experience evinces the marks it does, it because sensory experience is caused by mind-independent material objects. This might help explain why Berkeley is so intent on attacking the possibility that we can infer that our ideas are caused by mind-independent material objects.
In the Principles, Berkeley devotes even more energy to deflating this type of inference, and putting his arguments there and his arguments here together we can come up with a detailed, step by step, argument against mind- independent material objects as the cause of our ideas. In both works, Berkeley imagines three causal scenarios for the production of our ideas. On the first of these, mind-independent objects out in the world cause our ideas. This, of course, is the basic materialist line. On the second scenario, our ideas are caused directly by God. This is his own view. Finally, the third possibility is that God causes our ideas through the medium of mind-independent material objects. This is the line that Hylas tries to push once he has given up the straight materialist line.
In all, Berkeley has five considerations that he thinks show that the second of these scenarios is the best. The first four of these considerations are intended to work against the first scenario, and the fifth is intended to discredit the last scenario. Against the first scenario Berkeley has this to say: First, even if our ideas were caused by mind-independent material objects there would be no way to verify whether this were true or not, because there is no way to get out of our own ideas and check. When we, for instance, see something with our eyes, we can check whether our perception was correct by touching the object, or tasting it, and so on. But we have no analogous way to check on all of our sense faculties; we have nothing else to go one, to check these as a whole. So we have no way to ever determine whether mind-independent material objects cause our ideas.
Second, this scenario is actually completely meaningless, because we cannot attach any meaning to our idea of mind-independent material objects. The only way to form a meaningful idea, according to Berkeley, is to conjure up a precise image. And the only way to conjure up a precise image, according to anyone, would involve ascribing sensible qualities. We cannot have an image of something that has no color, size, shape, and so on. But Berkeley thinks he has already shown that all sensible qualities are mind-dependent, and so cannot belong to mind-independent material objects. So there is no way to conjure up a meaningful idea of a mind-independent material object, which means that this scenario too is utterly meaningless.
Third, even allowing that the concept of mind-independent material objects is not meaningless, they still could never cause our ideas. According to the materialists themselves, mind-independent material objects are inert, and are moved around by some sort of force, like energy or God. Inert things, though, cannot be causal agents. How could something inert cause anything?
Finally, even allowing that mind-independent material objects could be causal agents, what is even more inconceivable is that material objects could causally interact with immaterial objects. The notion is completely incoherent.