Filter Media Cleaning

The brown junk isn't the bacteria, it's fish poo, uneaten food etc. It does need to be washed off as it'll slow the flow through the filter.

White polyfibre could be filter wool or something else. Filter wool isn't a particularly good home for the bacteria and it doesn't wash well, but if your pad is something else it could wash fine. The only way to find out is to try, very carefully. Next time you do a water change, take the pad out and swoosh it gently in the water. Carefully rub the brown stuff off. You'll now have a feel for the pad. If it feels like it will disintegrate, don't do any more. If it feels quite firm, squeeze it gently.
The problem with replacing it, as you say, is that it affects the cycle by removing the vast majority of your bacteria. There will be some on all the surfaces of the tank including the filter casing but nowhere near enough to cope with the ammonia made by the fish. The best way to change media is to cut both old and new into three portions and change one portion a month till it's all changed.
If the white pad is the type that falls apart when washed, try not to damage it badly and change it as I mentioned, but replace it with something more substantial. If you can find some filter sponge, any make, that would fit, or could be cut up to fit, I would get that instead of a straight replacement.