I did exactly what you wrote but didn't see the effect you describe and what I saw yesterday was also different from what you describe.
I created a selection, set a filling color and filled the selection. I obtained a uniform blob with sharp boundaries as expected.
Then I did several somethings I don't remember (I think it involved changing color and probably resolution but not sure) and the previous blob with sharp boundaries suddenly completely changed.
Suppose that what I had before was a yellow blob on green background.
Now what I saw was that the blob transformed in a mixture of individual pixels. Some of them were yellow and some green. But as they were randomly (?) mixed, the sharp boundaries disappeared. The center of what was before a unicolor blob (e.g all pixels same color) had now more yellow than green pixels and the boundary had more green than yellow. The pixels had only 1 of both colors and not the unique color obtained by mixing green and yellow like before.
The overall effect was a blurred quasi smooth transition from green outside to yellow inside.
I remember that I then varied opacity from 100 to 0 and the mixture of green and yellow pixels gradually eliminated the yellow pixels untill at 0 none were left. But during the whole process the "pixel mixture aspect" didn't change.
I did another selection with the new color (black) and the aspect still was the pixelized blurred mixture.
I then left FT3 and when I next returned and went to overlay, I was only able to produce unicolor blobs with sharp boundaries. Even if the blurring pixelized effect was quite nice, I am unable to reproduce it.
It surely is nothing I saw mentionned in any document/tutorial/help function.
EDIT :
I think that I found it. Or at least something similar. When I start with low resolution (f.ex 100x100) and 100% opacity, I get big squares of uniform color more or less filling the selection. When I set now resolution to 5000x5000, the squares boundaries become completely blurred. When I then decrease the opacity, the whole becomes more blurred.