Overview

Annotations in Frame.io are represented as a serialized JSON blob of drawing objects, complete with colors, coordinates, and sizes. In the blob, all quotation marks apart from the outer bounding pair are escaped with forward slashes (e.g. \"key\":\"value\").

There are three basic shapes — arrow, rect, and line — as well as free-hand drawing, which is outside the scope of this guide, but follows the same basic principles of serialized coordinate data.

The coordinate system

All shapes are placed on a cartesian coordinate system with an upper-left origin, and a scale of 0 to 1 on each axis.

Basic shape data

All shapes are placed on the above plane with a combination of coordinates and style data. Each shape has some common attributes, and then each has a slightly different way of addressing the coordinate system.

All attributes for distance/width/height use the same 0 to 1 scale as the main coordinate grid itself, but are generally marked from the shape’s point of origin.

In other words, if a video is being displayed in a 640x480 viewport, x=1 will equal an absolute 640px in that context. This enables annotations to scale seamlessly while maintaining their relative position as viewing screen, size, and context shift.

Note: the origin of a rect is always the upper-left corner of the shape, regardless of drawing origin.

Common attributes

Attribute	Description
`tool`	arrow, rect, or line. States which shape is being made.
`color`	Six-digit hex color for the shape.
`size`	Pixel size for the shape’s line. For arrows, this refers only to the start of the line, as end width is determined programmatically.

Arrow

Attribute	Description
`x`	x-coordinate of the arrow’s origin.
`y`	y-coordinate of the arrow’s origin.
`w`	Difference in arrow’s y-axis position from start to end, relative to the arrow’s origin.
`h`	Difference in arrow’s x-axis position from start to end, relative to the arrow’s origin.

Rect

Attribute	Description
`x`	x-coordinate of the upper-left corner of the rectangle.
`y`	y-coordinate of the upper-left corner of the rectangle.
`w`	Width of the rectangle, relative to upper-left rectangle origin.
`h`	Height of the rectangle, relative to upper-left rectangle origin.
`ix`	The drawing x-origin of the rectangle. (Not required).
`iy`	The drawing y-origin of the rectangle. (Not required).
`radius`	Radius of the rectangle’s corner roundedness. Must be positive. Values above 150 are visually indistinguishable.

Line

Attribute	Description
`x1`	x-coordinate of the line’s origin.
`y1`	y-coordinate of the line’s origin.
`x2`	x-coordinate of the line’s terminus. Absolute value on the image plane.
`y2`	y-coordinate of the line’s terminus. Absolute value on the image plane.

Simple example: the coordinate plane

1 {"annotation": "[{\"tool\":\"line\",\"color\":\"#E74A3C\",\"size\":3,\"x1\":0.0028530670470756064,\"y1\":0.012680297987002695,\"x2\":0.9964336661911555,\"y2\":0.9903312727849104},{\"tool\":\"line\",\"color\":\"#34A3DB\",\"size\":3,\"x1\":0.004992867332382311,\"y1\":0.98779521318751,\"x2\":0.9942938659058488,\"y2\":0.012680297987002695}]}

1 {
2     "annotation": [
3         {
4             "tool": "line",
5             "color": "#E74A3C",
6             "size": 3,
7             "x1": 0.0028530670470756064,
8             "y1": 0.012680297987002695,
9             "x2": 0.9964336661911555,
10             "y2": 0.9903312727849104
11         },
12         {
13             "tool": "line",
14             "color": "#34A3DB",
15             "size": 3,
16             "x1": 0.004992867332382311,
17             "y1": 0.98779521318751,
18             "x2": 0.9942938659058488,
19             "y2": 0.012680297987002695
20         }
21     ]
22 }

Breakdown

1 {
2    "tool":"line",
3    "color":"#34A3DB",
4    "size":3,
5    "x1":0.004992867332382311,
6    "y1":0.98779521318751,
7    "x2":0.9942938659058488,
8    "y2":0.012680297987002695
9 }

1 {
2    "tool":"line",
3    "color":"#E74A3C",
4    "size":3,
5    "x1":0.0028530670470756064,
6    "y1":0.012680297987002695,
7    "x2":0.9964336661911555,
8    "y2":0.9903312727849104
9 }

Complex example: lots of boxes

1 {"annotation": "[{\"tool\":\"rect\",\"color\":\"#34A3DB\",\"size\":3,\"x\":0.008426966292134831,\"y\":0.02663337494798169,\"w\":0.0702247191011236,\"h\":0.10986267166042447,\"ix\":0.008426966292134831,\"iy\":0.02663337494798169,\"radius\":8},{\"tool\":\"rect\",\"color\":\"#1ABCA1\",\"size\":3,\"x\":0.8932584269662921,\"y\":0.0449438202247191,\"w\":0.08895131086142327,\"h\":0.35122763212650854,\"ix\":0.8932584269662921,\"iy\":0.0449438202247191,\"radius\":8},{\"tool\":\"rect\",\"color\":\"#E67422\",\"size\":3,\"x\":0.023408239700374533,\"y\":0.7873491468997087,\"w\":0.1254681647940075,\"h\":0.2047440699126093,\"ix\":0.023408239700374533,\"iy\":0.7873491468997087,\"radius\":8},{\"tool\":\"rect\",\"color\":\"#E74A3C\",\"size\":3,\"x\":0.6919475655430711,\"y\":0.5409904286308781,\"w\":0.29868913857677903,\"h\":0.4427798585101955,\"ix\":0.6919475655430711,\"iy\":0.5409904286308781,\"radius\":8},{\"tool\":\"arrow\",\"color\":\"#E74A3C\",\"size\":3,\"x\":0.25374531835205993,\"y\":0.23137744486059092,\"w\":0.24250936329588013,\"h\":0}]}"

1 {
2     "annotation": [
3         {
4             "tool": "rect",
5             "color": "#34A3DB",
6             "size": 3,
7             "x": 0.008426966292134831,
8             "y": 0.02663337494798169,
9             "w": 0.0702247191011236,
10             "h": 0.10986267166042447,
11             "ix": 0.008426966292134831,
12             "iy": 0.02663337494798169,
13             "radius": 8
14         },
15         {
16             "tool": "rect",
17             "color": "#1ABCA1",
18             "size": 3,
19             "x": 0.8932584269662921,
20             "y": 0.0449438202247191,
21             "w": 0.08895131086142327,
22             "h": 0.35122763212650854,
23             "ix": 0.8932584269662921,
24             "iy": 0.0449438202247191,
25             "radius": 8
26         },
27         {
28             "tool": "rect",
29             "color": "#E67422",
30             "size": 3,
31             "x": 0.023408239700374533,
32             "y": 0.7873491468997087,
33             "w": 0.1254681647940075,
34             "h": 0.2047440699126093,
35             "ix": 0.023408239700374533,
36             "iy": 0.7873491468997087,
37             "radius": 8
38         },
39         {
40             "tool": "rect",
41             "color": "#E74A3C",
42             "size": 3,
43             "x": 0.6919475655430711,
44             "y": 0.5409904286308781,
45             "w": 0.29868913857677903,
46             "h": 0.4427798585101955,
47             "ix": 0.6919475655430711,
48             "iy": 0.5409904286308781,
49             "radius": 8
50         },
51         {
52             "tool": "arrow",
53             "color": "#E74A3C",
54             "size": 3,
55             "x": 0.25374531835205993,
56             "y": 0.23137744486059092,
57             "w": 0.24250936329588013,
58             "h": 0
59         }
60     ]
61 }

Breakdown

With all that in mind, it’s helpful to lay out each element in the above image to develop a clearer picture of the annotation coordinate system. Note that the size and radius for each of the rectangles in this example are taking the Frame.io UI defaults.

Blue rect

1 {
2    "tool":"rect",
3    "color":"#34A3DB",
4    "size":3,
5    "x":0.008426966292134831,
6    "y":0.02663337494798169,
7    "w":0.0702247191011236,
8    "h":0.10986267166042447,
9    "ix":0.008426966292134831,
10    "iy":0.02663337494798169,
11    "radius":8
12 }

Green rect

1 {
2    "tool":"rect",
3    "color":"#1ABCA1",
4    "size":3,
5    "x":0.8932584269662921,
6    "y":0.0449438202247191,
7    "w":0.08895131086142327,
8    "h":0.35122763212650854,
9    "ix":0.8932584269662921,
10    "iy":0.0449438202247191,
11    "radius":8
12 }

Red rect

1 {
2    "tool":"rect",
3    "color":"#E74A3C",
4    "size":3,
5    "x":0.6919475655430711,
6    "y":0.5409904286308781,
7    "w":0.29868913857677903,
8    "h":0.4427798585101955,
9    "ix":0.6919475655430711,
10    "iy":0.5409904286308781,
11    "radius":8
12 }

Orange rect

1 {
2    "tool":"rect",
3    "color":"#E67422",
4    "size":3,
5    "x":0.023408239700374533,
6    "y":0.7873491468997087,
7    "w":0.1254681647940075,
8    "h":0.2047440699126093,
9    "ix":0.023408239700374533,
10    "iy":0.7873491468997087,
11    "radius":8
12 }

Red arrow

1 {
2    "tool":"arrow",
3    "color":"#E74A3C",
4    "size":3,
5    "x":0.25374531835205993,
6    "y":0.23137744486059092,
7    "w":0.24250936329588013,
8    "h":0
9 }