If anything add it during the primary after the main fermentation has taken over.  If you want it to be called "Vanilla" the vanilla should stand out in the beer and be noticable.  To get that you will want to add 2 vanilla beans, cut in half length wise, scrape the insides, then cut the beans into quarters.  I would then add it mabye after 3-4 days, just dump it right into the primary.  Then wait another 10 days and rack it off the beans.  You should get a nice aroma and flavour from that.

I think you may already know... 

 
 
 
This was discussed on the AHA website awhile ago. I copied/pasted some of the replies: 
1)I just split the bean and drop it in, no extra cutting or anything.  If you like, you could microwave it for about 15 seconds before you add it to the carboy.
2) I use 2 beans, one in the primary, and then, after I rack to secondary, I add the original bean and then a second one.  The first time I entered it in a contest I was actually told that there was "too much vanilla aroma and flavor".  The next year I entered it in the same competition and got very good comments.  The vanilla aged out a little.  I'm not sure how much to use for a beer but I would think that 1, or maybe 2, in the secondary, would be good.
3) My wife and I made a vanilla bean/cinnamon stick mead with our last batch. We used three vanilla beans extracted into a 750ml bottle of Stoli. The vanilla taste is VERY subtle, so if you do decide to use a bean, use more than 3. I don’t have a good number to give you, but perhaps 6 or better
4)I prefer Vanilla bean but it works best Steeped.  I would get a couple of nice plump vanilla beans, pound them down, slit the beans down the middle, peal open gently. Now I would steep this in with some priming sugar and add it late. You will get all the flavor and aroma.
5) I did a vanilla bean porter where I used a clone recipe for Sierra Nevada Porter (slightly modified) and then put 2 beans in my secondary.  A lot of my friend enjoyed this beer because it had a great taste of porter with a faint taste of the vanilla
6)I have added whole vanilla beans (2-3) to an oatmeal stout and it is incredible!  Simply cut open the beans and scrape the seeds into the secondary fermenter and let sit for about 2 weeks.  I find the vanilla flavor fades over time in the keg and is at its best about 1 month after kegging.  6)I have used Vanilla Beans before and had good success by making an extract.  First slit the beans open to expose the inside.  Then soak the beans in a small amount of Vodka for two weeks.  Remove the bean shell and dump the liquid into your secondary.  That should do it.  It might take two beans per 5gal. to make enough flavor to be noticeable.
7)I've only done this once with an Oatmeal Stout. I split the bean (one bean) then scraped the insides into about 1/4 Cup of Jim Beam, including the bean itself. The JB should sanitize the bean. I let this sit for a couple days to extract the vanilla flavor. Then dumped the cup into the keg.  Start small, one bean. If it isn't enough then try another (or one half bean). Don't go too heavy on the bean till you know how much to use. It's lots easier to add more than to take some out :-)
8)I had this same problem with vanilla when I used it like you did.  Great beer until about 2 weeks after I added the bean, then it went to medicinal/phenolic.  What I found out later is that whole vanilla beans can harbor beer spoiling bacteria.  Now when making a vanilla beer, I split open the bean and cut it into 1-inch strips.  I soak the beans in a couple ounces of vodka for about 10 days.  I put it in a small glass covered with Saran Wrap.  Then I dump the liquid and the beans in the secondary for conditioning.  I'll taste it periodically after the bean/vodka mix has been in the secondary for a couple weeks, and keg when it tastes good to me.