Help us shape the future of Go! Take the 2026 Go Developer Survey

siastorage

package module
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 27, 2026 License: MIT Imports: 36 Imported by: 0

Image README

Sia Storage SDK

The official Go SDK for storing and retrieving data on the Sia network.

For guides and additional resources, visit the developer portal. For detailed API documentation, see the Godocs.

Connecting to the Indexer

Before uploading or downloading data, your application must connect to an indexer. First, create a Builder with your application metadata, then walk the user through the approval flow:

builder := siastorage.NewBuilder("https://sia.storage", siastorage.AppMetadata{
	ID:          appID,                          // a persistent, randomly-generated 32-byte app ID
	Name:        "MyApp",                        // display name
	Description: "My first Sia application",     // short description
	LogoURL:     "https://my.app/logo.png",      // logo shown in the indexer UI
	ServiceURL:  "https://my.app",               // your application's homepage
})

// request a connection — the user must visit the returned URL to approve
responseURL, err := builder.RequestConnection(ctx)
if err != nil {
	log.Fatal("failed to request connection:", err)
}
fmt.Println("Approve the connection:", responseURL)

// block until the user approves or rejects
if err := builder.WaitForApproval(ctx); errors.Is(err, siastorage.ErrUserRejected) {
	log.Fatal("user denied the connection")
} else if err != nil {
	log.Fatal("failed to wait for approval:", err)
}

// derive an app key from a BIP-39 seed phrase and register it
mnemonic := siastorage.NewSeedPhrase() // generate once — store securely
client, err := builder.Register(ctx, mnemonic)
if err != nil {
	log.Fatal("failed to register:", err)
}
defer client.Close()

On subsequent launches, skip the approval flow and create the SDK directly with the previously derived app key:

builder := siastorage.NewBuilder("https://sia.storage", siastorage.AppMetadata{ID: appID})
client, err := builder.SDK(appKey)
if err != nil {
	log.Fatal("failed to create SDK:", err)
}
defer client.Close()

Uploading and Downloading Data

Once connected, you can upload and download files using the SDK:

// upload
obj := siastorage.NewEmptyObject()
f, _ := os.Open("path/to/src.dat")
defer f.Close()

if err := client.Upload(ctx, &obj, f); err != nil {
	log.Fatal("upload failed:", err)
}

// pin the object so the indexer tracks it
if err := client.PinObject(ctx, obj); err != nil {
	log.Fatal("pin failed:", err)
}

// download
out, _ := os.Create("path/to/dst.dat")
defer out.Close()

rc, err := client.Download(obj)
if err != nil {
	log.Fatal("download failed:", err)
}
defer rc.Close()

if _, err := io.Copy(out, rc); err != nil {
	log.Fatal("download failed:", err)
}

The Object returned by Upload contains the encryption key and slab metadata required to download the data later. After uploading, call PinObject to persist the object on the indexer. Applications should store the sealed object (via obj.Seal(appKey)) so it can be reopened later.

Packed Uploads

When uploading many small objects, UploadPacked combines them into shared slabs to reduce overhead:

packed, err := client.UploadPacked()
if err != nil {
	log.Fatal("failed to create packed upload:", err)
}
defer packed.Close()

for _, data := range smallObjects {
	if _, err := packed.Add(ctx, bytes.NewReader(data)); err != nil {
		log.Fatal("failed to add object:", err)
	}
}

objects, err := packed.Finalize(ctx)
if err != nil {
	log.Fatal("failed to finalize:", err)
}

// pin and download each object individually
for _, obj := range objects {
	client.PinObject(ctx, obj)
}
Optimizing Packed Uploads

Use Remaining() and Length() to monitor padding and decide when to finalize:

const paddingTarget = 0.05

for len(uploads) > 0 {
	packed, err := client.UploadPacked()
	if err != nil {
		log.Fatal(err)
	}

	for len(uploads) > 0 {
		if _, err := packed.Add(ctx, bytes.NewReader(uploads[0])); err != nil {
			log.Fatal(err)
		}
		uploads = uploads[1:]

		if packed.Length() == 0 {
			continue
		}
		if float64(packed.Remaining())/float64(packed.Length()) <= paddingTarget {
			break
		}
	}

	objects, err := packed.Finalize(ctx)
	if err != nil {
		log.Fatal(err)
	}

	for _, obj := range objects {
		client.PinObject(ctx, obj)
	}
	packed.Close()
}
Limitations
  • PackedUpload is not thread-safe; do not call Add concurrently
  • Empty objects are not supported and will return ErrEmptyObject
  • Once Finalize is called, subsequent Add calls return ErrUploadFinalized

Sharing Objects

Objects can be shared via time-limited URLs:

url, err := client.CreateSharedObjectURL(ctx, obj.ID(), time.Now().Add(24*time.Hour))
if err != nil {
	log.Fatal("failed to create shared URL:", err)
}

// anyone with the URL can download the object
var buf bytes.Buffer
rc, err := client.DownloadSharedObject(ctx, url)
if err != nil {
	log.Fatal("failed to download shared object:", err)
}
defer rc.Close()

if _, err := io.Copy(&buf, rc); err != nil {
	log.Fatal("failed to download shared object:", err)
}

Image Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrBuilderConsumed is returned when a [Builder] method is called
	// after the builder has already created an SDK. A builder is
	// single-use; create a new one to start another connection.
	ErrBuilderConsumed = errors.New("builder already consumed")

	// ErrNoConnectionRequest is returned by [Builder.WaitForApproval]
	// when called before [Builder.RequestConnection].
	ErrNoConnectionRequest = errors.New("no connection request")

	// ErrRequestExpired is returned by [Builder.WaitForApproval]
	// when the connection request has expired.
	ErrRequestExpired = errors.New("connection request expired")

	// ErrNotApproved is returned by [Builder.Register] when called
	// before [Builder.WaitForApproval] has successfully completed.
	ErrNotApproved = errors.New("connection not approved")

	// ErrUnauthorized is returned by [Builder.SDK] when the supplied
	// app key is not authorized by the indexer.
	ErrUnauthorized = errors.New("app key is not authorized")

	// ErrUserRejected is returned by [Builder.WaitForApproval] when
	// the user rejects the connection request.
	ErrUserRejected = app.ErrUserRejected
)
View Source 
var (
	// ErrEmptyObject is returned when trying to add an empty object.
	ErrEmptyObject = errors.New("empty object")
	// ErrUploadClosed is returned when trying to add an object to a closed upload.
	ErrUploadClosed = errors.New("upload is closed")
	// ErrUploadFinalized is returned when trying to add an object to an
	// already finalized upload.
	ErrUploadFinalized = errors.New("upload already finalized")
)
View Source 
var (
	// ErrNotEnoughShards is returned when not enough shards were
	// uploaded or downloaded to satisfy the minimum required shards.
	ErrNotEnoughShards = errors.New("not enough shards")

	// ErrNoMoreHosts is returned when there are no more hosts
	// available to attempt to upload a shard
	ErrNoMoreHosts = errors.New("no more hosts available")
)

Functions

func GenerateAppID 

func GenerateAppID() (id types.Hash256)

GenerateAppID generates a new random application ID.

func NewSeedPhrase 

func NewSeedPhrase() string

NewSeedPhrase generates a new seed phrase.

Types

type AppMetadata 

type AppMetadata struct {
	// ID is a unique identifier for an application.
	// It should be generated once and stay constant for
	// the lifetime of the app.
	//
	// Changing it will invalidate any existing app keys
	// and prevent access to associated data.
	//
	// It should be a randomly generated 32-byte value.
	// You can use GenerateAppID to create a new app ID.
	ID          types.Hash256
	Name        string
	Description string
	LogoURL     string
	ServiceURL  string
	CallbackURL string
}

AppMetadata contains metadata about an application. This metadata is provided during app registration and is used to identify the application to users.

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

A Builder helps connect an application to an indexer and initialize an SDK instance.

A Builder is single-use: once it has created an SDK, any further calls on the same Builder return ErrBuilderConsumed.

func NewBuilder 

func NewBuilder(indexerURL string, metadata AppMetadata) *Builder

NewBuilder creates a new Builder for connecting applications to the indexer.

A builder can only be used to create a single SDK instance. Methods called on a builder that has already created an SDK return ErrBuilderConsumed.

func (*Builder) Register 

func (b *Builder) Register(ctx context.Context, mnemonic string) (*SDK, error)

Register derives an application key from a BIP-39 seed phrase and registers it with the indexer.

This key should be stored securely by the application and never shared with anyone else. It can be regenerated using the same app ID, user account, and seed phrase.

It returns ErrBuilderConsumed if the builder has already created an SDK instance and ErrNotApproved if Builder.WaitForApproval has not yet returned successfully.

func (*Builder) RequestConnection 

func (b *Builder) RequestConnection(ctx context.Context) (string, error)

RequestConnection sends a request to connect an application to the indexer.

It returns a response URL that the user must visit to approve the request. The app should display the response URL to the user.

It returns ErrBuilderConsumed if the builder has already created an SDK instance.

func (*Builder) SDK 

func (b *Builder) SDK(appKey types.PrivateKey, opts ...Option) (*SDK, error)

SDK creates a new SDK instance using the given application key. If the key is not authorized, an error is returned.

It returns ErrBuilderConsumed if the builder has already created an SDK instance and ErrUnauthorized if the app key is not authorized by the indexer.

func (*Builder) WaitForApproval 

func (b *Builder) WaitForApproval(ctx context.Context) error

WaitForApproval waits for the user to approve the app connection request. The user must visit the response URL returned by Builder.RequestConnection to approve the request. It blocks until the request is approved, denied, or the context is cancelled.

It returns ErrUserRejected if the user denied the request and ErrBuilderConsumed if the builder has already created an SDK instance. Callers can branch on these using errors.Is.

type DownloadOption 

type DownloadOption func(*downloadOption)

A DownloadOption configures the download behavior

func WithDownloadHostTimeout 

func WithDownloadHostTimeout(timeout time.Duration) DownloadOption

WithDownloadHostTimeout sets the timeout for reading sectors from individual hosts. The default is 60 seconds.

func WithDownloadMaxBufferedChunks 

func WithDownloadMaxBufferedChunks(maxBufferedChunks int) DownloadOption

WithDownloadMaxBufferedChunks sets the maximum number of chunks held in memory. Download concurrency adapts to network conditions up to that ceiling. Each chunk uses up to 1 MiB; zero uses the default, roughly 10% of the memory available to the process.

func WithDownloadProgress 

func WithDownloadProgress(fn func(ShardProgress)) DownloadOption

WithDownloadProgress sets a callback that is invoked for shard downloads that complete successfully before the chunk download finishes, i.e. for up to MinShards successful shard downloads per chunk. Callers should keep the callback short or hand off work to a goroutine. The callback may be called concurrently.

func WithDownloadRange 

func WithDownloadRange(offset, length uint64) DownloadOption

WithDownloadRange sets the byte range to download from the object. The range is clamped to the object size: if offset+length exceeds the object size, only the available bytes are returned. If offset is at or beyond the end, or length is zero, the returned reader yields no data.

type Object 

type Object struct {
	// contains filtered or unexported fields
}

An Object represents a collection of slabs that can be used to access encrypted data. The master key is used to encrypt/decrypt the data and metadata, and should be kept secret.

It has no public fields to prevent accidental leakage of unencrypted data.

func NewEmptyObject 

func NewEmptyObject() Object

NewEmptyObject creates a new Object to use in [Upload].

func NewUnsafeObject 

func NewUnsafeObject(dataKey [32]byte, ss []slabs.SlabSlice) Object

NewUnsafeObject creates an Object from a data key and slabs. It can be used together with Object.UnsafeDataKey and Object.Slabs to reconstruct an object whose key and slabs were stored outside the indexer.

This is useful for interoperability with systems such as IPFS or LBRY, where an object's components are persisted separately and the object must be reconstructed from them.

Objects produced by SDK.Upload are guaranteed to be safe to reconstruct. Others, not so much. Here be dragons.

Invariants:

  • The data key must be the one that encrypted the slabs. A mismatched key fails silently: downloads succeed but return garbage.
  • Slab keys must never be reused. Reuse compromises encryption.
  • Each slab's version must match the version it was encrypted with. A mislabeled slab decrypts to garbage without error.
  • Each slab's offset and length must match how the data was encrypted: offset seeks the keystream, and slab order defines the object's byte stream. Wrong values silently corrupt or reorder the data.

The returned object has empty metadata and sets CreatedAt/UpdatedAt to time.Now.

func (*Object) CreatedAt 

func (o *Object) CreatedAt() time.Time

CreatedAt returns the time the object was created.

func (*Object) ID 

func (o *Object) ID() types.Hash256

ID returns the object's ID, which is a hash of its slabs.

func (*Object) Metadata 

func (o *Object) Metadata() json.RawMessage

Metadata returns a copy of the object's metadata.

func (*Object) Seal 

func (o *Object) Seal(appKey types.PrivateKey) SealedObject

Seal returns a SealedObject that can be safely serialized and shared.

func (*Object) Size 

func (o *Object) Size() uint64

Size returns the total size of the object in bytes.

func (*Object) Slabs 

func (o *Object) Slabs() []slabs.SlabSlice

Slabs returns a copy of the object's slabs.

func (*Object) UnsafeDataKey 

func (o *Object) UnsafeDataKey() [32]byte

UnsafeDataKey returns the key used to encrypt the object's data.

The data key alone decrypts the object's data. Never store it in plaintext and do not reuse it for new objects.

Prefer sealing the object with Object.Seal instead.

func (*Object) UpdateMetadata 

func (o *Object) UpdateMetadata(meta json.RawMessage)

UpdateMetadata updates the object's metadata.

func (*Object) UpdatedAt 

func (o *Object) UpdatedAt() time.Time

UpdatedAt returns the time the object was last updated.

type ObjectEvent 

type ObjectEvent struct {
	Key       types.Hash256
	Deleted   bool
	UpdatedAt time.Time
	Object    *Object
}

ObjectEvent represents a change to an object. If the object was deleted, Deleted is true and Object is nil.

type Option 

type Option func(*SDK)

An Option configures the SDK.

func WithLogger 

func WithLogger(log *zap.Logger) Option

WithLogger sets the logger for the SDK. The default behavior is to not log anything.

type PackedUpload 

type PackedUpload struct {
	// contains filtered or unexported fields
}

A PackedUpload allows multiple objects to be uploaded together in a single upload. This can be more efficient than uploading each object separately if the size of the objects is less than the optimal data size.

func (*PackedUpload) Add 

func (u *PackedUpload) Add(ctx context.Context, r io.Reader) (int64, error)

Add adds a new object to the upload. The data will be read until EOF and packed into the upload. The caller must call Finalize to get the resulting objects after all objects have been added.

func (*PackedUpload) Close 

func (u *PackedUpload) Close() error

Close closes the packed upload and releases any resources. The caller must always call Close to ensure proper cleanup.

func (*PackedUpload) Finalize 

func (u *PackedUpload) Finalize(ctx context.Context) ([]Object, error)

Finalize finalizes the upload and returns the resulting objects. This will wait for all slabs to be uploaded before returning. The resulting objects will contain the metadata needed to download the objects. The slabs are pinned to the indexer as they finish uploading, but the caller must still call PinObject for each returned object to save its metadata.

func (*PackedUpload) Length 

func (u *PackedUpload) Length() int64

Length returns the cumulative number of bytes written to the upload pipeline, including dead padding from errored reads.

func (*PackedUpload) OptimalDataSize 

func (u *PackedUpload) OptimalDataSize() int64

OptimalDataSize returns the data portion of a slab based on the number of data shards.

func (*PackedUpload) Remaining 

func (u *PackedUpload) Remaining() int64

Remaining returns the number of bytes remaining until reaching the optimal packed size. Adding objects larger than this will span multiple slabs. To minimize padding, prioritize objects that fit within the remaining size.

type SDK 

type SDK struct {
	// contains filtered or unexported fields
}

An SDK is a client for the indexd service.

func (*SDK) Account 

func (s *SDK) Account(ctx context.Context) (app.AccountResponse, error)

Account retrieves account information for the current app key.

func (*SDK) AppKey 

func (s *SDK) AppKey() types.PrivateKey

AppKey returns the app key used by the SDK.

It should be kept secret. Applications should store it securely to authenticate with the indexer.

func (*SDK) Close 

func (s *SDK) Close() error

Close closes the SDK and releases all resources.

func (*SDK) CreateSharedObjectURL 

func (s *SDK) CreateSharedObjectURL(ctx context.Context, objectKey types.Hash256, validUntil time.Time) (string, error)

CreateSharedObjectURL creates a URL that can be used to share the object until the given time. The URL contains the encryption key required to decrypt the object's data and metadata.

Sharing the URL allows anyone with the URL to read the object's data and metadata. They will not be able to modify the object or access any other objects in the account.

func (*SDK) DeleteObject 

func (s *SDK) DeleteObject(ctx context.Context, key types.Hash256) error

DeleteObject deletes the object with the given key from the indexer.

func (*SDK) Download 

func (s *SDK) Download(obj Object, opts ...DownloadOption) (io.ReadCloser, error)

Download returns an io.ReadCloser streaming the object's data. Closing the reader cancels the underlying download. Callers must always Close the returned reader to release resources.

func (*SDK) DownloadSharedObject 

func (s *SDK) DownloadSharedObject(ctx context.Context, sharedURL string, opts ...DownloadOption) (io.ReadCloser, error)

DownloadSharedObject returns an io.ReadCloser streaming a shared object's data. Closing the reader cancels the underlying download. Callers must always Close the returned reader to release resources.

func (*SDK) Object 

func (s *SDK) Object(ctx context.Context, objectKey types.Hash256) (Object, error)

Object retrieves the object with the given key.

func (*SDK) ObjectEvents 

func (s *SDK) ObjectEvents(ctx context.Context, cursor slabs.Cursor, limit int) ([]ObjectEvent, error)

ObjectEvents returns object events from the indexer, starting from the given cursor, up to the given limit. Unlike ListObjects, it preserves deletion events.

func (*SDK) PinObject 

func (s *SDK) PinObject(ctx context.Context, obj Object) error

PinObject saves the object metadata to the indexer, pinning any of its slabs that are not pinned yet.

Slabs uploaded through SDK.Upload or SDK.UploadPacked are pinned as they finish uploading, so pinning those objects takes a single request. Transient errors are not retried, calling PinObject again is safe and cheap.

func (*SDK) PruneSlabs 

func (s *SDK) PruneSlabs(ctx context.Context, opts ...api.URLQueryParameterOption) error

PruneSlabs removes all slabs on the account that are not associated with an object.

The indexer only considers slabs that were pinned more than api.DefaultSlabPruneCutoff ago. Pass api.WithBefore to override that cutoff.

func (*SDK) Upload 

func (s *SDK) Upload(ctx context.Context, obj *Object, r io.Reader, opts ...UploadOption) error

Upload uploads the data to hosts.

Appends the metadata of the slabs that were uploaded to the given object. The slabs are pinned to the indexer as they finish uploading, but the caller must still call PinObject to save the object metadata.

func (*SDK) UploadPacked 

func (s *SDK) UploadPacked(opts ...UploadOption) (*PackedUpload, error)

UploadPacked creates a new packed upload. This allows multiple objects to be packed together for more efficient uploads. The returned PackedUpload can be used to add objects and then finalized to get the resulting objects. A packed upload is not thread-safe.

type SealedObject 

type SealedObject struct {
	slabs.SealedObject
}

A SealedObject is an object that has been locked with an app key. It can be safely serialized and shared, but cannot be used to access the underlying data until it has been unlocked with the app key.

func (*SealedObject) Open 

func (so *SealedObject) Open(appKey types.PrivateKey) (Object, error)

Open decrypts the SealedObject using the given app key and returns an Object.

type ShardProgress 

type ShardProgress struct {
	HostKey    types.PublicKey
	SlabIndex  int
	ShardIndex int
	ShardSize  uint64
	Elapsed    time.Duration
}

A ShardProgress reports the result of a successfully completed shard upload or download.

type UploadOption 

type UploadOption func(*uploadOption)

An UploadOption configures the upload behavior

func WithRedundancy 

func WithRedundancy(dataShards, parityShards uint8) UploadOption

WithRedundancy sets the number of data and parity shards for the upload. The number of shards must be at least 2x redundancy: `(dataShards + parityShards) / dataShards >= 2`.

func WithUploadMaxBufferedSlabs 

func WithUploadMaxBufferedSlabs(maxBufferedSlabs int) UploadOption

WithUploadMaxBufferedSlabs sets the maximum number of fully encoded slabs held in memory. Upload concurrency adapts to network conditions up to that ceiling. Each slab uses (dataShards+parityShards) * 4 MiB; zero uses the default, roughly 10% of the memory available to the process.

func WithUploadProgress 

func WithUploadProgress(fn func(ShardProgress)) UploadOption

WithUploadProgress sets a callback that is invoked for each shard that completes uploading successfully. Callers should keep the callback short or hand off work to a goroutine. The callback may be called concurrently.

Image Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.