Yucky Dark Fluffy Growth On Wood In Dwarf Puffer Tank

I've spent an hour this evening picking dark fluffy bits off the wood in the dwarf puffer tank. I'm guessing it's some sort of fungal growth, but just wanted to check with some more experienced people.
I soaked all the wood for a few weeks before setting up the tank, but over the past week or so I've noticed the fluffy growth on the wood - the root style/mopani wood rather than the big pieces of drift wood. Previously I've seen the white kind of slimy stuff whilst soaking this wood, but this isn't like that.
Water parameters are 0, 0, <40, as always, but is there something I'm doing, or not doing, that is causing this? Apart from picking it off with the forceps is there anything else I can be doing to get rid of it. The fish don't seem bothered, but it is a problem for them?
Any advice appreciated.
Thanks. 

I've seen the brown fluffy stuff on wood in one or two of the tanks at the LFS, so I can always check with them when I visit at the end of the week if necessary. Was trying to avoid that for fear of leaving with more fish, or worse (depending on point of view) some little spanish ribbed newts or even something from the reptile section.
 

I have dark fluffy stuff on the wood in my 180 litre; I suspect it is a form of algae. I also have a rather attractive green growth on the wood which looks somewhat like small patches of mossball. Since mossballs are algae, that isn't a very far fetched idea.

The fluffy stuff of both colours got a lot worse in the 3 months the tank was fishless. I have assumed this is due to the imbalance in nutrients in the tank during this period - and all the whitespot medication I added to the tank.

I think I'll just take the wood out and give it a scrub.
Although it took me quite a long time to arrange the wood, so it might be a bit awkward, I think I'd rather take it out and deal with it than just keep picking bits off when I can.

@Matt  I was afraid that would be the problem. I've had a close look at the problem and it seems very similar to the white stuff, just a little darker. It certainly doesn't look like black brush algae, which I guess is some comfort.
How long did it take for nature to take its course in your tank? I don't tend to leave the tank lights on all day, I only turn them on in the tanks when I get home from work and turn them off when I'm going to bed, so usually around 5 hours a day. I've also realised that the  appeared after I used a liquid fertiliser in the tank. I didn't bother with it previously as I had put the JBL Aquabasic substrate in, followed by a layer of small gravel into the tank during set up, before planting. I figured that would be enough, especially as dwarf puffers produce a lot of waste, even though I'm over-filtering. I figure that adding a liquid fertiliser has provided way to many nutrients and caused the fluffy stuff to go wild. So far I've just been cleaning and doing water changes a bit more frequently than normal, but if there is anything else I can do please let me know.
Thanks. 

If this helps, in my tank, the white stuff on the first piece of redmoor wood took 10 weeks to completely disappear while the second piece of wood was boiled and had boiling water poured repeatedly over it but no white stuff was forthcoming (or, if it was, it was hidden under the anubias, java fern and java moss which were tied to it almost immediately).