That would be nothing for me. Can you determine with certainty what chemicals are used in your garden? If there is any uncertainty I would leave it. The spider is probably less interested in the smell than you and your visitors.
Hey there! I have asked myself the same question when I considered some broken terracotta pots myself as my hides. Whether you use them depends on how long they have been sitting in your garden due to possible exposure to chemicals, just as 8 legged has mentioned. If the pottery is lacquered or glazed (with a shiny surface), it would be less likely to absorb odd things, since ceramics are very porous and can absorb toxins. If the pottery item is matte and does not have a finish in any way, you might then have to consider its age and how long its been sitting out there.
The piece I selected in the past was a sherd of plain unfinished tan terracotta, however, as it began to gather moisture in an enclosure, the piece started to develop a crust of what appeared to be minerals along the edges. I was curious what that was, so I scraped a sample and took it to my work lab where I hit it with my spectrometer, a device which identifies chemical compounds. To my horror, I found fertilizers, pesticides, and even toxins accumulated from sitting in an urban air-polluted environment. I don't use anything in my garden, yet this example goes to say how absorbent ceramics can be. I tossed out the hide at once and promptly picked up a better piece of cork from my local pet store.
I hope this extended answer can help all the readers who have questions about pottery hides and up-cycling garden things in general for inverts!