How to: Upload (Advanced)

Introduction

Reliable asset uploading is the core function of every C2C integration. This guide provides advanced techniques and best practices for creating a robust, resilient, and efficient upload system that performs well even in challenging environments.

Prerequisites

If you haven’t already, please review the Implementing C2C: Setting Up guide before proceeding.

You’ll need the access_token obtained during the authentication and authorization process.

We’ll continue using the same test asset from the Basic Uploads guide.

Advanced Asset Parameters

When creating assets in Frame.io, you can use several advanced parameters to customize upload behavior. The offset parameter is particularly important for proper integration.

Offset - Handling Paused Devices

Providing an accurate offset value is critical. This parameter specifies when a piece of media was created and ensures your device doesn’t upload content that shouldn’t be shared. When a device is paused in Frame.io, the user is indicating that media created during the pause should not be uploaded. For more details, see our guide on pause functionality.

Additional Benefits of the offset Parameter

The offset parameter provides another significant advantage for organizing media withing Frame.io. When uploading content captured at an earlier date—for example, when a user selects a photo taken the previous week during playback—the offset parameter ensures this media appears in folders corresponding to its original capture date rather than the current upload date.

This chronological organization maintains a logical timeline in the Frame.io project structure. Without the offset parameter, historical media would incorrectly appear grouped with today’s content, potentially causing confusion for editors and other collaborators.

You may wish to provide users with a choice in this matter through your interface. If users prefer to organize all uploads by the current date regardless of when the media was captured, you can simply omit the offset parameter, as it defaults to 0 when not specified.

Our API design eliminates the need for your device to track pause status. Instead, when uploading a file, you indicate how many seconds ago the file was created. Our server compares this against pause windows and rejects the upload if it was created during a pause.

To demonstrate this feature, pause your device from the three-dot menu in the C2C Connections tab.

Now attempt to upload an asset:

$
{
>
curl -X POST https://api.frame.io/v2/devices/assets \
>
    --header 'Authorization: Bearer [access_token]' \
>
    --header 'Content-Type: application/json' \
>
    --header 'x-client-version: 2.0.0' \
>
    --data-binary @- <<'__JSON__' 
>
        {
>
            "name": "C2C_TEST_CLIP.mp4", 
>
            "filetype": "video/mp4", 
>
            "filesize": 21136250,
>
            "offset": 0
>
        }
>
__JSON__
>
} | python -m json.tool

You’ll receive this error:

1
{
2
    "code": 409,
3
    "errors": [
4
        {
5
            "code": 409,
6
            "detail": "The channel you're uploading from is currently paused.",
7
            "status": 409,
8
            "title": "Channel Paused"
9
        }
10
    ],
11
    "message": "Channel Paused"
12
}

If you unpause the device and retry with the same request, the asset will be created.

However, if the asset was created during the pause window, you need to set the offset to reflect when it was actually created:

$
{
>
curl -X POST https://api.frame.io/v2/devices/assets \
>
    --header 'Authorization: Bearer [access_token]' \
>
    --header 'Content-Type: application/json' \
>
    --header 'x-client-version: 2.0.0' \
>
    --data-binary @- <<'__JSON__' 
>
        {
>
            "name": "C2C_TEST_CLIP.mp4", 
>
            "filetype": "video/mp4", 
>
            "filesize": 21136250,
>
            "offset": 60
>
        }
>
__JSON__
>
} | python -m json.tool

This tells Frame.io the asset was created 60 seconds ago (during the pause), which properly triggers the Channel Paused error.

Accurate offset values are essential to prevent uploading sensitive content against the user’s wishes, including protected intellectual property, sensitive footage, or other restricted material.

Uploading to a Specific Channel

If your device has multiple channels, you can specify which one to use:

$
{
>
curl -X POST https://api.frame.io/v2/devices/assets \
>
    --header 'Authorization: Bearer [access_token]' \
>
    --header 'Content-Type: application/json' \
>
    --header 'x-client-version: 2.0.0' \
>
    --data-binary @- <<'__JSON__' 
>
        {
>
            "name": "C2C_TEST_CLIP.mp4", 
>
            "filetype": "video/mp4", 
>
            "filesize": 21136250,
>
            "offset": -10,
>
            "channel": 2
>
        }
>
__JSON__
>
} | python -m json.tool

If not specified, the default channel is 0. Most integrations won’t need to change this value.

Requesting a Custom Chunk Count

By default, Frame.io’s backend divides files into approximately 25MB chunks. For networks with high congestion, you might prefer smaller chunks. You can request a specific number of chunks with the parts parameter:

$
{
>
curl -X POST https://api.frame.io/v2/devices/assets \
>
    --header 'Authorization: Bearer [access_token]' \
>
    --header 'Content-Type: application/json' \
>
    --header 'x-client-version: 2.0.0' \
>
    --data-binary @- <<'__JSON__'
>
        {
>
            "name": "C2C_TEST_CLIP.mp4",
>
            "filetype": "video/mp4",
>
            "filesize": 21136250,
>
            "offset": 0,
>
            "parts": 4
>
        }
>
__JSON__
>
} | python -m json.tool

The response will include four upload URLs:

1
{
2
    ...
3
    "upload_urls": [
4
        "https://frameio-uploads-production.s3-accelerate.amazonaws.com/parts/[part-01-path]",
5
        "https://frameio-uploads-production.s3-accelerate.amazonaws.com/parts/[part-02-path]",
6
        "https://frameio-uploads-production.s3-accelerate.amazonaws.com/parts/[part-03-path]",
7
        "https://frameio-uploads-production.s3-accelerate.amazonaws.com/parts/[part-04-path]"
8
    ],
9
    ...
10
}

The chunk size will be:

1
math.ceiling(float(21136250) / float(4))
2
# 5284063 bytes

The last chunk will be 5,284,061 bytes (calculated as 21136250 - 5284063 * 3).

When requesting custom chunk counts, be aware of AWS S3 multipart upload limitations:

• Each part must be at least 5 MiB (5,242,880 bytes), except for the final part
• There can be no more than 10,000 parts

If your request violates these constraints, you’ll receive a 500: INTERNAL SERVER ERROR:

{
    "code": 500,
    "errors": [
        {
            "code": 500,
            "detail": "There was a problem with your request",
            "status": 500,
            "title": "Something went wrong"
        }
    ],
    "message": "Something went wrong"
}

Always verify your custom part count conforms to S3’s requirements.

Uploading Efficiently

C2C devices often operate in challenging network environments, so efficiency is crucial. Here are strategies to maximize throughput.

TCP Connection Reuse/Pooling

Establishing encrypted connections requires significant negotiation overhead. For efficient operation, reuse TCP connections when making multiple requests. Most HTTP libraries provide a Client or Session abstraction that maintains persistent connections.

The negotiation process for a new HTTPS connection includes cryptographic handshakes and certificate validation. By reusing connections, you only perform this overhead once rather than for each request.

To demonstrate connection reuse with curl, first create a new asset in Frame.io as described in the basic upload guide.

Next, split the file into separate chunks for testing:

$
head -c 10568125 ~/Downloads/C2C_TEST_CLIP.mp4 > "C2C_TEST_CLIP-Chunk01"
>
tail -c 10568125 ~/Downloads/C2C_TEST_CLIP.mp4 > "C2C_TEST_CLIP-Chunk02"

Now upload both chunks over a single TCP connection using curl’s --next parameter:

$
curl -X PUT https://frameio-uploads-production.s3-accelerate.amazonaws.com/parts/[part-1-path] \
>
        --include \
>
        --header 'content-type: video/mp4' \
>
        --header 'x-amz-acl: private' \
>
        --data-binary @C2C_TEST_CLIP-Chunk01 \
>
--next -X PUT https://frameio-uploads-production.s3-accelerate.amazonaws.com/parts/[part-2-path] \
>
        --include \
>
        --header 'content-type: video/mp4' \
>
        --header 'x-amz-acl: private' \
>
        --data-binary @C2C_TEST_CLIP-Chunk02

Compare this to separate connections:

$
curl -X PUT https://frameio-uploads-production.s3-accelerate.amazonaws.com/parts/[part-1-path] \
>
        --include \
>
        --header 'content-type: video/mp4' \
>
        --header 'x-amz-acl: private' \
>
        --data-binary @C2C_TEST_CLIP-Chunk01 \
>
&& curl -X PUT https://frameio-uploads-production.s3-accelerate.amazonaws.com/parts/[part-2-path] \
>
        --include \
>
        --header 'content-type: video/mp4' \
>
        --header 'x-amz-acl: private' \
>
        --data-binary @C2C_TEST_CLIP-Chunk02

In testing, connection reuse typically improves performance by 15-20% for sequential uploads.

Parallel Uploads

For even greater throughput, upload multiple chunks simultaneously:

$
curl -X PUT https://frameio-uploads-production.s3-accelerate.amazonaws.com/parts/[part-1-path] \
>
        --include \
>
        --header 'content-type: video/mp4' \
>
        --header 'x-amz-acl: private' \
>
        --data-binary @C2C_TEST_CLIP-Chunk01 \
>
& \
>
curl -X PUT https://frameio-uploads-production.s3-accelerate.amazonaws.com/parts/[part-2-path]\
>
        --include \
>
        --header 'content-type: video/mp4' \
>
        --header 'x-amz-acl: private' \
>
        --data-binary @C2C_TEST_CLIP-Chunk02 \
>
&

With sufficient bandwidth, parallel uploads complete in approximately the time of the slowest individual upload.

For optimal parallelism, a good rule of thumb is two concurrent uploads per CPU core. Exceeding this ratio can lead to resource contention and diminishing returns.

Combining Both Approaches

For maximum efficiency, combine connection pooling with parallel uploads. Create multiple processes, each using connection pooling for its own sequence of uploads:

$
curl -X PUT https://frameio-uploads-production.s3-accelerate.amazonaws.com/parts/[asset01-chunk01] \
>
        --include \
>
        --header 'content-type: video/mp4' \
>
        --header 'x-amz-acl: private' \
>
        --data-binary @C2C_TEST_CLIP-Chunk01 \
>
--next -X PUT https://frameio-uploads-production.s3-accelerate.amazonaws.com/parts/[asset01-chunk02] \
>
        --include \
>
        --header 'content-type: video/mp4' \
>
        --header 'x-amz-acl: private' \
>
        --data-binary @C2C_TEST_CLIP-Chunk02 \
>
& \
>
curl -X PUT https://frameio-uploads-production.s3-accelerate.amazonaws.com/parts/[asset02-chunk01] \
>
        --include \
>
        --header 'content-type: video/mp4' \
>
        --header 'x-amz-acl: private' \
>
        --data-binary @C2C_TEST_CLIP-Chunk01 \
>
--next -X PUT https://frameio-uploads-production.s3-accelerate.amazonaws.com/parts/[asset02-chunk02] \
>
        --include \
>
        --header 'content-type: video/mp4' \
>
        --header 'x-amz-acl: private' \
>
        --data-binary @C2C_TEST_CLIP-Chunk02 \
>
&

Tracking Upload Progress

Your integration must provide basic progress indication to users. Chunk-level granularity is acceptable—for a three-chunk upload, progress might increment from 0% → 33% → 66% → 100% as each chunk completes.

Finer-grained progress reporting depends on your HTTP library’s capabilities. Contact our team if you need guidance on implementing more detailed progress tracking.

Uploading Reliably

For robust error handling, review our errors guide. The following sections assume you’ve implemented the error handling strategies described there.

Creating a production-quality uploader requires additional considerations beyond handling individual request errors.

Creating an Upload Queue

In real-world scenarios, your device may generate media faster than it can upload, or it might experience extended connection interruptions. Implementing a queuing system separates media creation from upload management.

Consider a two-queue architecture:

1. A media queue for registering local files with Frame.io
2. A chunk queue for uploading individual file chunks

Here’s a simplified implementation:

1
# Where we are going to queue new files.
2
FILE_QUEUE = Queue()
3
4
# Where we are going to queue new chunks.
5
CHUNK_QUEUE = Queue()
6
7
# The http session that will handle TCP 
8
# connection pooling for us.
9
HTTP_SESSION = http.Session()
10
11
def take_picture():
12
    """Snaps a picture for the user."""
13
14
    image = MY_DEVICE.capture()
15
    file_path = MY_DEVICE.write_image(image)
16
    FILE_QUEUE.add(file_path)
17
18
def task_register_assets():
19
    """
20
    Pulls snapped pictures from the FILE_QUEUE, registers
21
    with Frame.io, and adds the chunks to the CHUNK_QUEUE.
22
    """
23
    while True:
24
        # Get the latest file added to the queue and register 
25
        # a C2C Asset for it.
26
        new_file = FILE_QUEUE.get()
27
        asset = c2c.crete_asset_for_file(HTTP_SESSION, new_file)
28
29
        # Calculate the size for each chunk
30
        chunk_size = c2c.calculate_chunk_size(asset, new_file)
31
32
        # Create a message for each chunk with it's parameters 
33
        # and add it to the 
34
        # queue.
35
        chunk_start = 0
36
        for chunk_url in asset.upload_urls:
37
            message = {
38
                "file_path": new_file,
39
                "chunk_url": chunk_url,
40
                "chunk_start": chunk_start,
41
                "chunk_size": chunk_size,
42
            }
43
44
            # Put the message in the queue and 
45
            CHUNK_QUEUE.put(message)
46
            chunk_start += chunk_size
47
48
def task_upload_chunk():
49
    """Takes a chunks and uploads them."""
50
51
    while True:
52
        info = CHUNK_QUEUE.get()
53
        c2c.upload_chunk(HTTP_SESSION, info)
54
55
def launch_upload_tasks():
56
    """Lauches our Frame.io upload tasks."""
57
    # Create a list to hold all of our tasks.
58
    tasks = list()
59
60
    # Create one task for registering assets.
61
    asset_task = run_task_in_thread(task_register_assets)
62
    tasks.append(asset_task)
63
64
    # Create 2 tasks per CPU core for uploading chunks.
65
    for _ in range(0, GET_CPU_COUNT() * 2):
66
        chunk_task = run_task_in_thread(task_upload_chunk)
67
        tasks.append(chunk_task)
68
69
    # Run these tasks until shutdown
70
    run_forever(tasks)

Persistent Queuing Across Power Cycles

The in-memory queue approach works well while the device remains powered on, but what happens if power is lost before uploads complete? To create a truly resilient integration, we need to ensure the device can resume from where it left off after restarting.

This requires persisting the queue state to storage between power cycles. An embedded database such as SQLite provides an excellent foundation for this functionality.

Your persistent queue implementation should support these key operations:

• Adding newly created files to the upload queue
• Tracking when assets are successfully created in Frame.io
• Recording when asset creation fails due to errors
• Storing file chunk information for upload tasks
• Retrieving the next chunk to be uploaded
• Marking chunks as successfully uploaded
• Logging chunk upload failures
• Providing file status information for user display

Here’s how we might adapt our previous example to use a persistent storage system:

1
# Our persistence layer for queuing uploads, potentially using SQLite 
2
# or another embedded database
3
C2C_UPLOAD_STORE = NewC2CUploadStore()
4
5
# HTTP session for connection pooling
6
HTTP_SESSION = http.Session()
7
8
def take_picture():
9
    """Captures an image and adds it to the upload queue."""
10
    image = MY_DEVICE.capture()
11
    file_path = MY_DEVICE.write_image(image)
12
13
    # Register the file with our persistent store
14
    C2C_UPLOAD_STORE.add_file(file_path)
15
16
def task_register_assets():
17
    """
18
    Processes files from persistent storage and registers
19
    them with Frame.io for upload.
20
    """
21
    while True:
22
        # Get the next available file from our store
23
        file_record = C2C_UPLOAD_STORE.get_file()
24
25
        try:
26
            # Register the asset with Frame.io
27
            asset = c2c.create_asset_for_file(HTTP_SESSION, file_record)
28
            chunk_size = c2c.calculate_chunk_size(asset, file_record)
29
30
            # Create entries for each chunk in our persistent store
31
            chunk_start = 0
32
            for chunk_url in asset.upload_urls:
33
                message = {
34
                    "file_path": file_record,
35
                    "chunk_url": chunk_url,
36
                    "chunk_start": chunk_start,
37
                    "chunk_size": chunk_size,
38
                }
39
                
40
                C2C_UPLOAD_STORE.new_chunk(message)
41
                chunk_start += chunk_size
42
                
43
        except BaseException as error:
44
            # Record the error in our persistent store
45
            C2C_UPLOAD_STORE.file_asset_create_error(file_record, error)
46
        else:
47
            # Mark the asset as successfully created
48
            C2C_UPLOAD_STORE.file_asset_created(file_record)
49
50
def task_upload_chunk():
51
    """Uploads individual file chunks from the persistent queue."""
52
    while True:
53
        # Get the next chunk, marking it as "in progress" to prevent 
54
        # other tasks from processing it simultaneously
55
        chunk_record = C2C_UPLOAD_STORE.get_chunk()
56
57
        try:
58
            c2c.upload_chunk(HTTP_SESSION, chunk_record)
59
        except BaseException as error:
60
            # Record the error for potential retry
61
            C2C_UPLOAD_STORE.chunk_error(chunk_record, error)
62
        else:
63
            # Mark successful completion
64
            C2C_UPLOAD_STORE.chunk_success(chunk_record)
65
66
def launch_upload_tasks():
67
    """Launches Frame.io upload processing tasks."""
68
    tasks = []
69
70
    # Asset registration task
71
    asset_task = run_task_in_thread(task_register_assets)
72
    tasks.append(asset_task)
73
74
    # Multiple parallel chunk upload tasks
75
    worker_count = GET_CPU_COUNT() * 2
76
    for _ in range(worker_count):
77
        chunk_task = run_task_in_thread(task_upload_chunk)
78
        tasks.append(chunk_task)
79
80
    # Run indefinitely
81
    run_forever(tasks)

With this persistent storage approach, your integration becomes resilient to power interruptions. When the device restarts, it simply continues processing from its last saved state. This architecture also provides the foundation for implementing more advanced features, like error tracking and stalled upload detection.

Tracking Upload Errors

A robust upload system must carefully track errors. After retrying an operation using the strategies in the errors guide, record these failures in your persistence store. This allows your system to:

1. Deprioritize problematic uploads to prevent them from blocking the entire queue
2. Provide accurate status information to users
3. Enable administrative intervention for persistent issues

When a fatal error occurs, mark the item to prevent unnecessary retry attempts.

Managing Stalled Uploads

Implement safeguards against indefinitely stalled uploads. Set a maximum duration (e.g., 30 minutes) after which a chunk upload task should be terminated and restarted. This prevents scenarios where all upload workers become blocked by non-responsive operations.

Recovering From Silent Failures

System crashes, power loss, or process termination can prevent normal error reporting. When retrieving items from your queue, record the checkout time. If an item remains in the “in progress” state beyond a reasonable threshold (e.g., 30 minutes) without reporting success or failure, automatically return it to the available pool for processing by another worker.

Mitigating Poisoned Uploads

A “poisoned” queue item consistently fails due to inherent problems with the data or environment. If these items continuously requeue, they can effectively block your entire upload system. Consider these strategies for handling such cases:

• After multiple failures, deprioritize the item so newer content can proceed
• Track both explicit errors and the number of processing attempts
• Follow connection and authorization best practices to distinguish between transient environmental issues and intrinsic file problems
• Implement escalating retry limits (e.g., retry individual operations 10 times within each of 3 job attempts, for 30 total attempts)
• Provide a user interface for manually resetting problematic uploads once environmental issues are resolved

Poisoned uploads can result from:

• Corrupted file data causing I/O errors
• Catastrophic process failures that prevent error reporting
• Normally retriable errors triggered by permanent underlying conditions

Retry After System Restart

Before permanently abandoning problematic uploads, flag them for one final retry after the next system restart. This addresses cases where uploads fail due to temporary system state issues with memory, drivers, or resource allocation. If an upload continues to fail after a clean restart, you can more confidently mark it as permanently problematic.

Clearing Your Queue

Remember to remove unavailable files from your queue. When media is physically removed or files are deleted, purge corresponding entries from your upload queue to prevent unnecessary errors.

Importantly, you must clear your upload queue when connecting to a new project. Media queued for one project should never appear in another. When a user pairs the device with a different project, verify whether the project has changed and, if so, completely clear the existing queue.

Next Steps

We encourage you to contact our team with any questions and proceed to the advanced uploads guide. We look forward to supporting your integration progress.