Wayfire has decoration plugins and even a shadow plugin available. The shadow plugin works great, for square decorations at least. Adding rounded corners with shadows that hug them would be a nice addition. So, I set out to do that, and the results even surprised me.

I started hacking on pixdecor plugin, first porting it to work with latest wayfire, then adding compute shader support for decoration animations, patterns and overlay effects. Out of all the things, none of the effects had a need to render beyond the decoration area, into the shadow region. This basically meant making the rendering area larger, to have a place to scribble in some shadows. The concept is simple but implementing it required changing the code very carefully in many different places. Even after getting it right, changing some settings might make the decoration the wrong size or some other problem (like crashing). After shaking out the bugs for the shadow padding, it was time to shade some shadows.

I knew I wanted simple code that got straight to the point. After poking around a bit on shadertoy and not finding what I wanted, I decided to write the shader from scratch. Here’s the path I ended up with. First, the decoration is cleared using a clear color or otherwise filled in with an effect shader. Then, the overlay shader is applied. The overlay has eight conditionals, one for each side and one for each corner. For the sides, it measures the distance from the edge to the pixel color being written and mixes transparent with the shadow color, depending on how far away it is. The corner is the same recipe but it picks a single point in the corner and uses it to measure the corner and shadow radius. This means that even though the corners are rounded, the shadows follow the curvature. The other nice thing about the overlay is that is automatically anti-aliases the edges of the rounded corners by mixing, instead of just discarding pixels or other method that leaves jagged corner edges.

All of this, and everything is configurable in real time, from the border size to the corner radius and shadow radius, color, etc. It took some careful coding to make everything work together but in the end, it is working nicely, with great performance. Now, for the video: