Perlite As A Filter Medium

Has anyone tried using Perlite as a filter medium? (yes - Perlite from the local Garden Centre).
I saw a guy recommending it on a You Tube video, stating it keeps the water beautifully clear.
It's an inert material, so does no harm to the fish.
I'm trying it and it is doing a good job so far.

The fiddliness was my first thought too. I use perlite for my African violets (mix potting compost 2:1 with perlite or vermiculite) and it is very light and fluffy. The only way it could be used, in my opinion, is in a fine mesh bag.
I presume it crumbles quite easily as now I'm getting to the bottom of the bag I'm finding a lot of dust instead of tiny balls of white. This would be my concern about using it as a filter medium - will it crumble to dust when wet, like it does when dry, and slowly disappear especially after cleaning the goo out? I would experiment with some in a bag in a tub of water before putting it in the filter.

Some good observations there.
I forgot to mention that I put it in a fine mesh bag before placing it in the filter.
Also, before filling the bag, I washed out the dust and smaller particles with some tank water, through a colander.
The perlite has to go into a bag because it is not very dense and tends to float.