Skip to content

Commit 3c00a39

Browse files
authored
feat(storage-azure): client uploads with chunkLargeFiles can now support files larger than 5gb (#17319)
v3 port of #17318 In v4 this is enabled by default, in v3 this is behind a flag because it's potentially breaking as CORS needs additional headers setup. ## What Adds opt-in support for uploading files larger than ~5GB directly from the client with the Azure storage adapter. A new `clientUploads.chunkLargeFiles` option routes browser uploads through the Azure Blob SDK, which splits the file into blocks instead of sending it in a single request. ## Why Client uploads (`clientUploads: true`) previously sent each file to Azure in one `Put Blob` request. Azure caps a single `Put Blob` at ~5GB, so any larger file failed outright — which defeats the purpose of client uploads, since they exist specifically to move large files that can't go through the server/proxy body limits. The Blob SDK's block API (`Put Block` + `Put Block List`) is Azure's supported path for large blobs. It splits the file into blocks, uploads them in parallel, and commits at the end, raising the effective ceiling to the block-blob maximum (~190TiB) while also improving throughput on large transfers. ## Behaviour - **Default is unchanged.** With `clientUploads: true`, uploads still use the original single-request path and the same CORS requirements. Existing setups behave exactly as before — this is not a breaking change. - **Large-file support is opt-in.** Set `clientUploads: { chunkLargeFiles: true }` to switch to the SDK block-upload path for files over ~5GB. ```ts azureStorage({ clientUploads: { chunkLargeFiles: true, }, // ... }) ``` `chunkLargeFiles` routes client uploads through the Azure Blob SDK, which sends extra `x-ms-*` headers and issues CORS preflight (OPTIONS) requests. The storage account's CORS rules must therefore allow: - Methods: OPTIONS, PUT (GET/HEAD too if reading blobs in-browser) - Allowed headers: `*` (or at minimum `x-ms-*,content-type,content-length`) - Exposed headers: `*` Without these, block uploads fail in the browser with "RestError: Failed to fetch".
1 parent f14722a commit 3c00a39

8 files changed

Lines changed: 361 additions & 54 deletions

File tree

‎docs/upload/storage-adapters.mdx‎

Lines changed: 27 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,8 @@ keywords: uploads, images, media, storage, adapters, s3, vercel, google cloud, a
88

99
Payload offers additional storage adapters to handle file uploads. These adapters allow you to store files in different locations, such as Amazon S3, Vercel Blob Storage, Google Cloud Storage, and more.
1010

11-
| Service | Package |
12-
| -------------------- | ----------------------------------------------------------------------------------------------------------------- |
11+
| Service | Package |
12+
| -------------------- | ---------------------------------------------------------------------------------------------------------------- |
1313
| Vercel Blob | [`@payloadcms/storage-vercel-blob`](https://github.com/payloadcms/payload/tree/3.x/packages/storage-vercel-blob) |
1414
| AWS S3 | [`@payloadcms/storage-s3`](https://github.com/payloadcms/payload/tree/3.x/packages/storage-s3) |
1515
| Azure | [`@payloadcms/storage-azure`](https://github.com/payloadcms/payload/tree/3.x/packages/storage-azure) |
@@ -216,7 +216,26 @@ pnpm add @payloadcms/storage-azure
216216

217217
- Configure the `collections` object to specify which collections should use the Azure Blob adapter. The slug _must_ match one of your existing collection slugs.
218218
- When enabled, this package will automatically set `disableLocalStorage` to `true` for each collection.
219-
- When deploying to Vercel, server uploads are limited with 4.5MB. Set `clientUploads` to `true` to do uploads directly on the client. You must allow CORS PUT method to your website.
219+
- When deploying to Vercel, server uploads are limited with 4.5MB. Set `clientUploads` to `true` to do uploads directly on the client.
220+
221+
- Client uploads send each file in a single request, which Azure caps at ~5GB. To upload larger files, set `clientUploads: { chunkLargeFiles: true }` — this uploads through the Azure Blob SDK, which splits the file into blocks and raises the limit to Azure's block-blob maximum.
222+
223+
<Banner type="warning">
224+
**`chunkLargeFiles` and CORS:** With `chunkLargeFiles: true`, the Azure Blob SDK
225+
sends additional `x-ms-*` headers, so the browser issues a CORS preflight. Your
226+
storage account's CORS rules must allow the `OPTIONS` and `PUT` methods **and**
227+
those headers. Configure a CORS rule on the Blob service (Storage account →
228+
**Resource sharing (CORS)**):
229+
230+
| Field | Value |
231+
| --------------- | -------------------------------------------------------- |
232+
| Allowed origins | Your site origin (e.g. `https://example.com`) |
233+
| Allowed methods | `GET,PUT,OPTIONS` (`HEAD` if reading in-browser) |
234+
| Allowed headers | `*` (or at minimum `x-ms-*,content-type,content-length`) |
235+
| Exposed headers | `*` |
236+
| Max age | `3600` |
237+
238+
</Banner>
220239

221240
```ts
222241
import { azureStorage } from '@payloadcms/storage-azure'
@@ -452,13 +471,13 @@ This plugin is configurable to work across many different Payload collections. A
452471

453472
## Collection-specific options
454473

455-
| Option | Type | Description |
456-
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
474+
| Option | Type | Description |
475+
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
457476
| `adapter` \* | [Adapter](https://github.com/payloadcms/payload/blob/3.x/packages/plugin-cloud-storage/src/types.ts#L49) | Pass in the adapter that you'd like to use for this collection. You can also set this field to `null` for local development if you'd like to bypass cloud storage in certain scenarios and use local storage. |
458-
| `disableLocalStorage` | `boolean` | Choose to disable local storage on this collection. Defaults to `true`. |
459-
| `disablePayloadAccessControl` | `true` | Set to `true` to disable Payload's Access Control. [More](#payload-access-control) |
477+
| `disableLocalStorage` | `boolean` | Choose to disable local storage on this collection. Defaults to `true`. |
478+
| `disablePayloadAccessControl` | `true` | Set to `true` to disable Payload's Access Control. [More](#payload-access-control) |
460479
| `generateFileURL` | [GenerateFileURL](https://github.com/payloadcms/payload/blob/3.x/packages/plugin-cloud-storage/src/types.ts#L67) | Override the generated file URL with one that you create. |
461-
| `prefix` | `string` | Set to `media/images` to upload files inside `media/images` folder in the bucket. |
480+
| `prefix` | `string` | Set to `media/images` to upload files inside `media/images` folder in the bucket. |
462481

463482
## Prefix Composition
464483

‎packages/storage-azure/README.md‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,32 @@ pnpm add @payloadcms/storage-azure
1414

1515
- Configure the `collections` object to specify which collections should use the Azure Blob Storage adapter. The slug _must_ match one of your existing collection slugs.
1616
- When enabled, this package will automatically set `disableLocalStorage` to `true` for each collection.
17-
- When deploying to Vercel, server uploads are limited with 4.5MB. Set `clientUploads` to `true` to do uploads directly on the client. You must allow CORS PUT method to your website.
17+
- When deploying to Vercel, server uploads are limited with 4.5MB. Set `clientUploads` to `true` to do uploads directly on the client.
18+
19+
### Large client uploads (files over ~5GB)
20+
21+
By default, client uploads (`clientUploads: true`) send each file in a single request, which Azure caps at ~5GB. To upload larger files, set `chunkLargeFiles: true`:
22+
23+
```ts
24+
azureStorage({
25+
// ...
26+
clientUploads: {
27+
chunkLargeFiles: true,
28+
},
29+
})
30+
```
31+
32+
This uploads through the Azure Blob SDK, which splits the file into blocks (`Put Block` + `Put Block List`) and raises the limit to Azure's block-blob maximum (~190TiB).
33+
34+
Because the SDK sends additional `x-ms-*` headers, the browser issues a CORS preflight, so your storage account's CORS rules must allow the `OPTIONS` and `PUT` methods **and** those headers. Configure a CORS rule on the Blob service (Storage account → **Resource sharing (CORS)**):
35+
36+
| Field | Value |
37+
| --------------- | -------------------------------------------------------- |
38+
| Allowed origins | Your site origin (e.g. `https://example.com`) |
39+
| Allowed methods | `GET,PUT,OPTIONS` (add `HEAD` if reading in-browser) |
40+
| Allowed headers | `*` (or at minimum `x-ms-*,content-type,content-length`) |
41+
| Exposed headers | `*` |
42+
| Max age | `3600` |
1843

1944
```ts
2045
import { azureStorage } from '@payloadcms/storage-azure'
Lines changed: 15 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -1,58 +1,32 @@
11
'use client'
22
import { createClientUploadHandler } from '@payloadcms/plugin-cloud-storage/client'
3-
import { formatAdminURL } from 'payload/shared'
43

5-
export const AzureClientUploadHandler = createClientUploadHandler({
4+
import { handleAzureUpload } from './handleAzureUpload.js'
5+
6+
export type AzureClientUploadHandlerExtra = {
7+
chunkLargeFiles: boolean
8+
}
9+
10+
export const AzureClientUploadHandler = createClientUploadHandler<AzureClientUploadHandlerExtra>({
611
handler: async ({
712
apiRoute,
813
collectionSlug,
914
docPrefix,
15+
extra: { chunkLargeFiles },
1016
file,
1117
serverHandlerPath,
1218
serverURL,
1319
updateFilename,
1420
}) => {
15-
const endpointRoute = formatAdminURL({
21+
return handleAzureUpload({
1622
apiRoute,
17-
path: serverHandlerPath,
23+
chunkLargeFiles,
24+
collectionSlug,
25+
docPrefix,
26+
file,
27+
serverHandlerPath,
1828
serverURL,
29+
updateFilename,
1930
})
20-
const response = await fetch(endpointRoute, {
21-
body: JSON.stringify({
22-
collectionSlug,
23-
docPrefix,
24-
filename: file.name,
25-
mimeType: file.type,
26-
}),
27-
credentials: 'include',
28-
method: 'POST',
29-
})
30-
31-
const {
32-
docPrefix: sanitizedDocPrefix,
33-
filename: sanitizedFilename,
34-
url,
35-
} = (await response.json()) as {
36-
docPrefix: string
37-
filename?: string
38-
url: string
39-
}
40-
41-
if (sanitizedFilename && sanitizedFilename !== file.name) {
42-
updateFilename(sanitizedFilename)
43-
}
44-
45-
await fetch(url, {
46-
body: file,
47-
headers: {
48-
'Content-Length': file.size.toString(),
49-
'Content-Type': file.type,
50-
// Required for azure
51-
'x-ms-blob-type': 'BlockBlob',
52-
},
53-
method: 'PUT',
54-
})
55-
56-
return { prefix: sanitizedDocPrefix }
5731
},
5832
})
Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
import { beforeEach, describe, expect, it, vi } from 'vitest'
2+
3+
const uploadDataMock = vi.fn()
4+
const blockBlobClientMock = vi.fn(function () {
5+
return { uploadData: uploadDataMock }
6+
})
7+
8+
vi.mock('@azure/storage-blob', () => ({
9+
BlockBlobClient: blockBlobClientMock,
10+
}))
11+
12+
import { handleAzureUpload } from './handleAzureUpload.js'
13+
14+
const serverURL = 'https://example.com'
15+
const apiRoute = '/api'
16+
const serverHandlerPath = '/storage-azure-generate-signed-url' as const
17+
const signedURL = 'https://account.blob.core.windows.net/container/file.png?sig=abc'
18+
19+
const createFile = () => new File([new Uint8Array([1, 2, 3])], 'file.png', { type: 'image/png' })
20+
21+
const mockSignedURLResponse = (body: Record<string, unknown>, ok = true) => {
22+
const fetchMock = vi.fn().mockResolvedValue({
23+
json: () => Promise.resolve(body),
24+
ok,
25+
})
26+
vi.stubGlobal('fetch', fetchMock)
27+
return fetchMock
28+
}
29+
30+
const invoke = (overrides: Partial<Parameters<typeof handleAzureUpload>[0]> = {}) =>
31+
handleAzureUpload({
32+
apiRoute,
33+
chunkLargeFiles: true,
34+
collectionSlug: 'media',
35+
file: createFile(),
36+
serverHandlerPath,
37+
serverURL,
38+
updateFilename: vi.fn(),
39+
...overrides,
40+
})
41+
42+
describe('handleAzureUpload', () => {
43+
beforeEach(() => {
44+
vi.clearAllMocks()
45+
vi.unstubAllGlobals()
46+
uploadDataMock.mockResolvedValue(undefined)
47+
})
48+
49+
describe('shared behavior', () => {
50+
it('should request a signed URL from the server handler', async () => {
51+
const fetchMock = mockSignedURLResponse({ docPrefix: 'docs', url: signedURL })
52+
53+
await invoke({ docPrefix: 'docs' })
54+
55+
const [calledURL, init] = fetchMock.mock.calls[0]!
56+
expect(calledURL).toBe(`${serverURL}${apiRoute}${serverHandlerPath}`)
57+
expect(init.method).toBe('POST')
58+
expect(JSON.parse(init.body)).toEqual({
59+
collectionSlug: 'media',
60+
docPrefix: 'docs',
61+
filename: 'file.png',
62+
mimeType: 'image/png',
63+
})
64+
})
65+
66+
it('should call updateFilename when the server returns a sanitized filename', async () => {
67+
mockSignedURLResponse({ docPrefix: 'docs', filename: 'file-1.png', url: signedURL })
68+
const updateFilename = vi.fn()
69+
70+
await invoke({ updateFilename })
71+
72+
expect(updateFilename).toHaveBeenCalledWith('file-1.png')
73+
})
74+
75+
it('should not call updateFilename when the filename is unchanged', async () => {
76+
mockSignedURLResponse({ docPrefix: 'docs', filename: 'file.png', url: signedURL })
77+
const updateFilename = vi.fn()
78+
79+
await invoke({ updateFilename })
80+
81+
expect(updateFilename).not.toHaveBeenCalled()
82+
})
83+
84+
it('should return the sanitized doc prefix', async () => {
85+
mockSignedURLResponse({ docPrefix: 'sanitized-docs', url: signedURL })
86+
87+
const result = await invoke()
88+
89+
expect(result).toEqual({ prefix: 'sanitized-docs' })
90+
})
91+
92+
it('should throw when the signed URL request fails, before any upload', async () => {
93+
mockSignedURLResponse({}, false)
94+
95+
await expect(invoke()).rejects.toThrow()
96+
97+
expect(uploadDataMock).not.toHaveBeenCalled()
98+
expect(blockBlobClientMock).not.toHaveBeenCalled()
99+
})
100+
})
101+
102+
describe('chunkLargeFiles: true (SDK block upload)', () => {
103+
it('should upload via BlockBlobClient.uploadData rather than a raw PUT', async () => {
104+
const fetchMock = mockSignedURLResponse({ docPrefix: 'docs', url: signedURL })
105+
const file = createFile()
106+
107+
await invoke({ chunkLargeFiles: true, file })
108+
109+
expect(blockBlobClientMock).toHaveBeenCalledWith(signedURL)
110+
expect(uploadDataMock).toHaveBeenCalledTimes(1)
111+
112+
const [uploadedFile, options] = uploadDataMock.mock.calls[0]!
113+
expect(uploadedFile).toBe(file)
114+
expect(options.blockSize).toBeGreaterThan(0)
115+
expect(options.concurrency).toBeGreaterThan(0)
116+
117+
// Only the signed-URL request should hit fetch; no raw PUT to the blob URL.
118+
expect(fetchMock).toHaveBeenCalledTimes(1)
119+
})
120+
121+
it('should pass the file content type to the blob headers', async () => {
122+
mockSignedURLResponse({ docPrefix: 'docs', url: signedURL })
123+
124+
await invoke({ chunkLargeFiles: true })
125+
126+
const [, options] = uploadDataMock.mock.calls[0]!
127+
expect(options.blobHTTPHeaders).toEqual({ blobContentType: 'image/png' })
128+
})
129+
130+
it('should propagate errors from uploadData', async () => {
131+
mockSignedURLResponse({ docPrefix: 'docs', url: signedURL })
132+
uploadDataMock.mockRejectedValue(new Error('block upload failed'))
133+
134+
await expect(invoke({ chunkLargeFiles: true })).rejects.toThrow('block upload failed')
135+
})
136+
})
137+
138+
describe('chunkLargeFiles: false (single PUT, default/legacy)', () => {
139+
it('should upload via a single raw PUT with the BlockBlob header', async () => {
140+
const fetchMock = mockSignedURLResponse({ docPrefix: 'docs', url: signedURL })
141+
const file = createFile()
142+
143+
await invoke({ chunkLargeFiles: false, file })
144+
145+
// 1st fetch = signed-URL POST, 2nd = the raw PUT to the blob URL
146+
expect(fetchMock).toHaveBeenCalledTimes(2)
147+
const [putURL, putInit] = fetchMock.mock.calls[1]!
148+
expect(putURL).toBe(signedURL)
149+
expect(putInit.method).toBe('PUT')
150+
expect(putInit.body).toBe(file)
151+
expect(putInit.headers['x-ms-blob-type']).toBe('BlockBlob')
152+
expect(putInit.headers['Content-Type']).toBe('image/png')
153+
})
154+
155+
it('should not use the Azure SDK', async () => {
156+
mockSignedURLResponse({ docPrefix: 'docs', url: signedURL })
157+
158+
await invoke({ chunkLargeFiles: false })
159+
160+
expect(blockBlobClientMock).not.toHaveBeenCalled()
161+
expect(uploadDataMock).not.toHaveBeenCalled()
162+
})
163+
})
164+
})

0 commit comments

Comments
 (0)