New Sketchpane: Alchemancy Mark III

New Sketchpane: Alchemancy Mark III

This is the proposed plan for the new Sketchpane / drawing system / painting engine / sketching mechanism, however in this doc, it'll just be referred to as the Sketchpane.

What is Alchemancy Mark III?

Alchemancy Mark II is the current name of the Sketchpane engine of Storyboarder. Even though, as said above, the Sketchpane would be referred to as Sketchpane, we must talk about the most important aspect of a cutting edge, next generation Sketchpane: the pretentious marketing name.

Storyboarder paints fast. Really fast. For years, Storyboarder has set the bar for performance in creative applications with Alchemancy Mark II, Storyboarder’s exceptionally fluid drawing and painting engine. So how do you improve on already industry-leading performance? Meet Alchemancy Mark III, the next-generation painting engine built with technologies like Javascript, XML, text editors, and probably some PHP. Alchemancy Mark III has been designed to extract the full power of computer to provide creative professionals with incredible color depth, fluid and accurate painting, and more performance than ever before.

If you think we stole that from the Procreate website, no we didn't. We wrote it first. There is probably something wrong with the Github date stamp.

Alchemancy, at its digitally emotional core, means the divination of your ideas into universal elixir of life and beauty. Every stroke you make is seminal. From god's gloryhole, to your storyboards, Alchemancy Mark III transforms the movement of your mortal dirty dickbeaters into a symphony of eternal color, texture, form, and feeling.

But from here on out we'll just call it a Sketchpane.

What is the Sketchpane?

In simple terms: it's a component that holds a canvas that you can draw onto. There is one canvas inside of the Sketchpane. You can draw on the canvas, but you can't draw outside of it in the Sketchpane. You can change zoom into the canvas and pan around in the Sketchpane. You can also resize the sketchpane itself (Imagine resizing the window).

What is the canvas?

To avoid ambiguity, the canvas is a general term and shouldn't be confused with Javascript's Canvas Object.

The canvas is the document that contains the imagery. The canvas can also contain layers. The layers are stacked on top of each other in a specific order and composited according to their blend mode. https://threejs.org/examples/webgl_materials_blending.html

What is a layer?

Each layer is an image (a bitmap, a texture). Layers should, but don't necessarily have to, be the same size as the canvas. Layer's dimensions should be in the form of pixels – not points, picas, or inches or anything dumb like that.

What happens when I draw?

When you draw, you actually aren't actually drawing directly to a layer on the canvas. The first thing is we determine which layer we want to be drawing on. Then we create a new temporary layer/texture directly above that layer. As we draw the stroke, it is drawn to that temporary layer. When we lift the pointer, we are essentially ending the stroke. When this happens, we draw the temporary layer to the layer we meant to be drawing on. We clear the temporary layer.

Why draw to a temporary layer?

It's very possible as you're drawing that we might want to change the way the stroke looks. Imagine you were drawing a line. You were drawing, but then the app realized you wanted a straight line, it could straighten the line you already drew. I know sounds crazy, but it's not.

So how is a stroke drawn?

First, it's important to know that a stroke is not driven directly by a pointer input. Yes, a pointer input can result in a stroke being rendered, however, the pointer input is first processed, and that processor tells the Sketchpane what to do. So imagine you were drawing like a crazy person. Crazy zig zags and such. The app may want to smooth your inputs into smoother strokes, so it doesn't look jagged when it renders.

At a super high level, a stoke can be thought of as thousands of rectangular brush textures along a line.

A stroke is collection of line segments. A line segment is a collection of brush node plots.

A stroke can also change while it's being drawn. WHAT? WHY? OK. Remember when the iPad came out, and everyone was jizzing all over themselves that there is no lag? Well, that's just impossible. There's always lag. Photoshop and Sketchbook pro have a 2 frame delay before they draw what you input, so they can smooth the line and draw it to the screen. But the iPad does something kinda smart, they predict where you will be next, and they are mostly right, unless they aren't, and it fixes itself so fast you don't even notice it. It's the fixing itself part that is dynamic.

So every frame, as the stroke is being updated, the drawing is being updated. This results in many rectangular brush textures as their own polygons, living on the screen. That's cool. But what if my line is like, REALLY REALLY REALLY LONG? Won't that get super slow? That could be like a billion nodes?

Optimization!

Yes - too many nodes is not cool. Those newly added nodes could still be moving around. But the older nodes, at the beginning of the line segment, they are likely static, and will not change. THEREFORE, you can draw those nodes to the temporary texture. THEREFORE, you can have a super duper long line and it's always drawing the last bits to the texture so there aren't that many dynamic brush nodes composited in real time. Pretty sweet, eh?

Whats in a brush node?

As you are drawing a stoke, segments are being created of nodes. A node contains information like:

Property
Position
Pressure
Tilt / Rotation
External Size Parameter
External Opacity Parameter
Brush

So what are brushes?

A brush is a description of how the brush node should be rendered. It's a collection of attributes. Imagine a very simple brush. It might need only two attributes: size and color. If you specify a brush that is 20px in diameter and yellow, you can draw a fat yellow line. If you specify 3px and blue, you will have a distinctly different skinny blue line.

Brushes can have many more very specific attributes, like how is the opacity of the brush affected by the tilt and pressure of the pen? There are more brush attributes listed below.

How did the old Sketchpane work?

Oh, you mean Alchemancy Mark II? It kinda worked as described above. The only problem is how it was built. The layers were rendered at HTML Canvas Objects on top of each other and composited by the DOM. This is OK for small canvases on websites, but for huge canvases that are also being scaled by the DOM, this is really super slow. Also, we used HTML Canvases to render the brush strokes. This wasn't that slow, but we couldn't render brushes like we will propose below.

We did our best to profile the performance and eek out every last optimization we could. But on slower machines, drawing sucks. It's super slow. Users report "jaggies", "lag", and other non specific words that developers hate.

What is the solution? OpenGL.

Most computers have a graphics card capable of doing lots of graphics specific operations very quickly and concurrently. For example, using DOM rendering, we struggle to display more than 3 layers and maintain 60 fps. Using OpenGL, we can display 10+ layers at 4K squared resolution, and not miss a single frame.

OpenGL Tests:

1. Can layers work? 10+ 4K layers composited on top of each other with various blend types maintaining 60fps.
2. Can long strokes work? Multiple (1000+) textures rendered to a renderTexture (layer) maintaining 60fps.
3. Can strokes be shaders? Multiple (1000+) textures as fragment shaders rendered to a renderTexture maintaining 60fps.

I think the answers to all these questions are yes!

OpenGL Implementation

The Sketchpane is a GL context where all the layers are stacked in order.

Layers are renderTextures on a polygon rendered in a specific order.

Brush nodes are polygons with a universal fragment shader in which parameters are set by the definition of the brush.

Brush Node Properties

Brushes have 3 different properties: Node properties, External Brush Properties, and Brush Properties.

Node Properties

Node properties come from the stroke information (Imagine the stylus):

Properties	Notes
Position X, Y	Where the node is placed in X, Y
Pressure	Pressure of the stylus if applicable (Used by the brush definition)
Tilt / Rotation	Tilt / Rotation of the stylus if applicable (Used by the brush definition)
Time	Time that the node was created. Used to calculate speed.
Speed	Calculated in by the input processor can be used by the brush definition

External Brush Properties

External Brush Properties come from the external UI. Imagine an interface where you have distinct brushes you can use, but for each of the brushes, you can pick a color.

Properties	Notes
Color	Hex value definition for the base color for the brush
Size	Diameter in pixels. However, this is passed into the brush definition only as an attribute. (The size can change i.e. pressure.)
Opacity	Float from 0-1. However, this is passed into the brush definition only as an attribute. (The size can change i.e. pressure.)

Brush Properties

The following properties define a brush preset. This is where the magic happens.

At its simplest, a brush has a grayscale image that is used as an alpha for the brush, and a grayscale image that is used as an alpha for the grain texture.

So when a brush node is drawn, a node class loads the brush properties and figures out where the polygon should be placed, the size, the rotation, and shader is started for the polygon, and all the attributes are passed into the shader as properties.

It takes the color property uses it as the base of the image. It uses the brush image as an alpha channel of the brush. The alpha is also multiplies by the grain image. Based on the location of the brush node, the grain image is offset, so that as the nodes are being drawn, it appears that the grain texture is that of the canvas, and not of the brush. (It's an illusion.) I wrote a simple shader: https://www.shadertoy.com/view/4tjyRK

Images

Property	Value	Default	Notes
Brush Image	Texture2D (Grayscale, Square)	Circle	Used as the alpha channel of the brush
Brush Image Rotation	0,90,180,270	0	Can rotate the image for the shader
Brush Image Invert	Boolean	False	Invert the values
Grain Image	Texture2D (Grayscale, Square, Repeating)	Solid	Used as the alpha channel of the paper grain
Grain Image Rotation	0,90,180,270	0	Can rotate the image for the shader
Grain Image Invert	Boolean	False	Invert the values

Stroke

Property	Value	Default	Notes
Spacing	[0-100%]	5%	0 is very close together, 100 is spaced out
Smoothing	[0-100%]	0%	0 is no smoothing, 100 is delayed smoothing
Jitter	[0-100%]	0%	0 is right on line, 100% is randomly scattered
Falloff	[0-100%]	0%	0 is totally solid, 100% fades out the line quickly
Stroke Taper Start	[0-100%]	0%	% of the line where the start taper is affected
Stroke Taper End	[0-100%]	100%	% of the line where the end taper is affected
Stroke Taper Opacity	[0-100%]	0%	% the opacity is decreased for the taper
Stroke Taper Size	[0-100%]	100%	% the size is for the taper

Shape

Property	Value	Default	Notes
Scatter	[0-100%]	0%	Scatter of the grain
Rotation	[-100-100%]	0%	Rotation of the alpha shape
Azimuth	Boolean	False	Change the size and rotation based on the angle of the stylus

Grain

Property	Value	Default	Notes
Movement	[0-100%]	100%	% the grain is offset as the brush moves. 0 static. 100 rolling. 100 is like paper.
Scale	[0-100%]	30%	Scale of the grain texture. 0 super tiny, 100 super zoomed.
Zoom	[0-100%]	0%	% Scale of the grain texture by the brush size.
Rotation	[-100-100%]	0%	% Rotation grain rotation is multiplied by rotation
Filtered	Boolean	True	Is the texture antialiased or nearest neighbor?

Dynamics

Property	Value	Default	Notes
Rendering	Normal, Glazed, Wet Mix	Normal	TBD
Opacity from speed	[-100% - 100%]	0%	How much opacity is affected based on speed
Opacity Jitter	[0 - 100%]	0%	Randomness of opacity
Size from Speed	[-100% - 100%]	0%	How much size is affected based on speed
Size Jitter	[0 - 100%]	0%	Randomness of size

Stylus Settings

Property	Value	Default	Notes
Pressure Opacity	[-100% - 100%]	0%	% Pressure affects opacity
Pressure Size	[-100% - 100%]	0%	% Pressure affects size
Tilt Angle	[0-90]	0	Degrees angle altered by the tilt
Tilt Opacity	[0-100%]	0%	% opacity altered by the tilt
Tilt Gradiation	[0-100%]	0%	How it changes as tilt happens, curve compression
Tilt Size	[0-100%]	0%	% size altered by the tilt

General

Property	Value	Default	Notes
Name	String	''	Name of the brush preset
Blend Mode	[blend mode]	Copy	https://threejs.org/examples/webgl_materials_blending.html
Size Limit Max	[0-100%]	100%	Max brush size settable in UI
Size Limit Min	[0-100%]	0%	Min brush size settable in UI
Opacity Limit Max	[0-100%]	100%	Max brush opacity settable in UI
Opacity Limit Min	[0-100%]	0%	Min brush opacity settable in UI
Default Layer	Int	0	Does brush draw to own layer?

Erasing

Guides

Operations

Lasso Selections

Straight lines