Skip to content

Module github.com/bide-ai/bide/anchor/gcs ​

Go module github.com/bide-ai/bide/anchor/gcs; all modules.

anchor/gcs ​

github.com/bide-ai/bide/anchor/gcs · pkg.go.dev · Stability: experimental

Package gcs is an audit.Anchor that writes every signed tree head as an object in a Google Cloud Storage bucket whose retention policy is locked (a WORM store), and a Reader that reads the heads back and verifies them.

Constants

  • const DefaultPrefix = "bide-anchor/": DefaultPrefix is the object name prefix used when Config.Prefix is empty.
  • const Locked = "Locked": Locked is the per-object retention mode that cannot be shortened or removed.
  • const MinRetention = 24 * time.Hour: MinRetention is the shortest retention New accepts.
  • const ObjectFormat = "bide.audit.gcs-anchor.v1": ObjectFormat is the format of an anchored object's body.

Variables

  • var WriterRole string: WriterRole is the role for the application that publishes heads.
  • var WriterObjectRetentionRole string: WriterObjectRetentionRole is WriterRole plus storage.objects.setRetention, which a create that sets a per-object retention needs (Config.ObjectRetention); verified against a real bucket, where the plain writer role was refused with a 403 naming that permission.
  • var ReaderRole string: ReaderRole is the role for an auditor or monitor that only reads heads.
  • var HoldReleaserRole string: HoldReleaserRole is the role for the identity that releases temporary holds.
  • var ErrUnavailable = errors.New("gcs: anchor unavailable"): ErrUnavailable wraps an error that means Cloud Storage could not be reached or did not answer in time (a transport error, a context that ended, a throttle or a server error).
  • var ErrWeak: ErrWeak wraps the error of a stored object whose lock, retention or encryption is weaker than the configuration asked for.
  • var ErrConflict: ErrConflict wraps the error of a name that already holds different bytes than the head being published.
  • var ErrNotFound = errors.New("gcs: no anchored head"): ErrNotFound is returned by the Reader when a partition holds no head.
  • var ErrVanished: ErrVanished is returned by the Reader when a generation that was listed cannot be read: it was deleted between the listing and the read (at the end of its retention) or is hidden.
  • var ErrWrongProject: ErrWrongProject wraps (with agent.ErrConfig) the error of a bucket that another project owns than the one expected: a bucket name that was deleted and created again by someone else, say.
  • var ErrBadSignature: ErrBadSignature wraps (with ErrObject) the error of an object that is a well-formed head of its partition whose signature does not verify under the log key: a forgery, a head signed by another key (a rotated one, say), or a re-signed one.
  • var ErrEquivocation: ErrEquivocation wraps the error of two validly signed heads of one tree and size that commit to different roots: the log key signed two histories.
  • var ErrObject: ErrObject wraps the error of an object that is not a well-formed anchored head: a key and a body that disagree, a body that is not an anchored object, or a head that does not verify.
  • var ErrExists = errors.New("gcs: object exists"): ErrExists is Create's answer to a name that holds a live object (HTTP 412).
  • var ErrNoObject = errors.New("gcs: no such object"): ErrNoObject is the answer to a name or generation that does not exist (HTTP 404).
  • var ErrDenied = errors.New("gcs: permission denied"): ErrDenied is the answer to a call the caller's identity may not make (HTTP 401 or 403).
  • var MethodPermissions: MethodPermissions maps each Store method to the IAM permissions it needs.
  • var ObjectRetentionPermissions = []string{"storage.objects.setRetention"}: ObjectRetentionPermissions are the permissions a Create that sets a per-object retention needs in addition to MethodPermissions["Create"].

Functions

  • func Layout() anchorobj.Layout: Layout returns this backend's layout: its object format, partition tag and error values.
  • func LoadDir(dir string, v audit.Verifier) ([]audit.SignedTreeHead, error): LoadDir reads a downloaded set of anchored objects (for example the output of "gcloud storage cp -r" of the anchor prefix) and verifies each against v.
  • func RolePermissions(role string) ([]string, error): RolePermissions returns the permissions a role document (WriterRole and the others) grants.

type Anchor ​

go
type Anchor struct {
	// contains filtered or unexported fields
}

Anchor writes signed tree heads to Cloud Storage under a locked retention policy.

Constructors and functions

  • func New(ctx context.Context, cfg Config) (*Anchor, error): New validates the bucket and returns an Anchor.

Methods

  • func (a *Anchor) Publish(ctx context.Context, runID string, sth audit.SignedTreeHead) error: Publish writes sth as an object and reads it back; it returns nil only when the object is stored, retained as configured, encrypted with the configured key and equal to sth.

type BucketInfo ​

go
type BucketInfo struct {
	ProjectNumber uint64

	RetentionPeriod time.Duration

	RetentionLocked bool
	Versioning      bool

	DefaultKMSKey string

	UniformAccess bool

	PublicAccessPrevented bool

	ObjectRetention bool

	SoftDeleteDuration time.Duration
}

BucketInfo is a bucket's configuration, in the terms the anchor checks.

type Config ​

go
type Config struct {
	Store Store

	ExpectedProjectNumber uint64

	Prefix string

	Retention time.Duration

	ObjectRetention bool

	LegalHold bool

	KMSKey string

	EncryptionNone bool

	OnWarn func(error)

	RefuseSuspectIDs bool

	Now func() time.Time
}

Config configures an Anchor.

type CreateOptions ​

go
type CreateOptions struct {
	KMSKey        string
	TemporaryHold bool
	Retention     *ObjectRetention
}

CreateOptions are the attributes Create writes with an object.

type Finding ​

go
type Finding struct {
	Key     string
	Version string
	Problem string
}

Finding is something Audit found wrong with the bucket's lock state.

Methods

  • func (f Finding) String() string: String is the finding as one line.

type ObjectInfo ​

go
type ObjectInfo struct {
	Name       string
	Generation int64
	Size       int64
	Created    time.Time

	Noncurrent time.Time
	CRC32C     uint32

	KMSKey string

	RetentionExpiry time.Time

	Retention     *ObjectRetention
	TemporaryHold bool
	EventHold     bool
}

ObjectInfo is what Cloud Storage reports about one generation of an object.

type ObjectRetention ​

go
type ObjectRetention struct {
	Mode        string
	RetainUntil time.Time
}

ObjectRetention is a per-object retention configuration.

type Reader ​

go
type Reader struct {
	// contains filtered or unexported fields
}

Reader reads anchored heads back from Cloud Storage and verifies them.

Constructors and functions

  • func NewReader(ctx context.Context, cfg ReaderConfig) (*Reader, error): NewReader returns a Reader.

Methods

  • func (r *Reader) Audit(ctx context.Context, kind, runID string, now time.Time) (Report, error): Audit reads every generation of every object of a tree and reports its retention state: Heads are the verified heads, and Findings lists every noncurrent generation (something overwrote or deleted the live object), every generation no retention covers (it can be deleted), every generation whose retention has ended (it can now be deleted), every per-object retention that is not Locked, every object not encrypted with the configured key (when ReaderConfig.KMSKey is set), every name with more than one generation (the name is never rewritten by this package), every object that is not a verified head (and, when its name claims a size larger than the largest verified head, that too), and two verified heads of one size with different roots, and each bucket-level protection that is gone (the retention policy absent, unlocked or under the minimum; versioning, uniform bucket-level access or public access prevention off; with ReaderConfig.KMSKey, another default key).
  • func (r *Reader) Head(ctx context.Context, kind, runID string, size int) ([]audit.SignedTreeHead, error): Head returns the heads anchored for the tree at exactly the given size (more than one only when two processes anchored one run at the same size, with the same root), or ErrNotFound; two with different roots are an error wrapping ErrEquivocation.
  • func (r *Reader) Heads(ctx context.Context, kind, runID string) ([]audit.SignedTreeHead, error): Heads returns every head anchored for the tree, in increasing size (heads of one size in name order), each verified, with the skipping of Latest and its ErrVanished.
  • func (r *Reader) Latest(ctx context.Context, kind, runID string) (audit.SignedTreeHead, error): Latest returns the head of the largest size anchored for the tree of the given kind (audit.TreeJournal or audit.TreeEvents) and run or trail ID.

type ReaderConfig ​

go
type ReaderConfig struct {
	Store Store

	ExpectedProjectNumber uint64

	Prefix string

	KMSKey string

	Retention time.Duration

	OnSkip func(Finding)

	Verifier audit.Verifier
}

ReaderConfig configures a Reader.

type Report ​

go
type Report struct {
	Heads    []audit.SignedTreeHead
	Findings []Finding
}

Report is the result of Audit.

type Store ​

go
type Store interface {
	Bucket(ctx context.Context) (BucketInfo, error)

	Create(ctx context.Context, name string, body []byte, o CreateOptions) (ObjectInfo, error)

	Stat(ctx context.Context, name string, generation int64) (ObjectInfo, error)

	Read(ctx context.Context, name string, generation int64, limit int64) ([]byte, ObjectInfo, error)

	List(ctx context.Context, prefix string, versions bool, fn func(ObjectInfo) bool) error
}

Store is the part of Cloud Storage this package uses, for one bucket.

Constructors and functions

  • func NewStore(c *storage.Client, bucket string) Store: NewStore returns the Store for a bucket of a Cloud Storage client.

anchor/gcs/gcstest ​

github.com/bide-ai/bide/anchor/gcs/gcstest · pkg.go.dev · Stability: experimental

Package gcstest is an in-memory gcs.Store that models the parts of Cloud Storage the anchor relies on: object generations, create-only writes (ifGenerationMatch=0), a bucket retention policy that can be locked, noncurrent generations under versioning, temporary holds, per-object retention and Cloud KMS keys.

Constants

  • const DefaultKMSKey: DefaultKMSKey is the default Cloud KMS key of a new Fake.
  • const ProjectNumber = 123456789012: ProjectNumber is the project number of a new Fake's bucket.

Variables

  • var ErrRetained: ErrRetained is what a delete of a retained or held generation returns.

type Fake ​

go
type Fake struct {
	// contains filtered or unexported fields
}

Fake is the in-memory bucket.

Constructors and functions

  • func New() *Fake: New returns a bucket that is what the anchor needs: a locked 30-day retention policy, versioning, uniform access, enforced public access prevention and DefaultKMSKey as its default key.

Methods

  • func (f *Fake) Advance(d time.Duration): Advance moves the clock forward.
  • func (f *Fake) AlterNextCreate(fn func(*gcs.ObjectInfo)): AlterNextCreate changes the attributes of the generation the next Create writes, as a bucket that stores an object under other terms than were asked for would.
  • func (f *Fake) Bucket(context.Context) (gcs.BucketInfo, error): Bucket implements gcs.Store.
  • func (f *Fake) Calls() []string: Calls returns the Store methods called so far, in order.
  • func (f *Fake) Configure(fn func(*gcs.BucketInfo)): Configure changes the bucket's configuration (to make a bucket the anchor must refuse).
  • func (f *Fake) Create(_ context.Context, name string, body []byte, o gcs.CreateOptions) (gcs.ObjectInfo, error): Create implements gcs.Store.
  • func (f *Fake) DeleteGeneration(name string, generation int64) error: DeleteGeneration permanently deletes one generation.
  • func (f *Fake) DeleteLive(name string) error: DeleteLive deletes the live generation of name.
  • func (f *Fake) FailNext(method string, err error, times int): FailNext makes the next times calls of method (Bucket, Create, Stat, Read or List) return err without running.
  • func (f *Fake) Generations(name string) []gcs.ObjectInfo: Generations returns the infos of every generation of name, oldest first.
  • func (f *Fake) Inject(info gcs.ObjectInfo, body []byte) gcs.ObjectInfo: Inject adds a generation exactly as given (a foreign writer's object: an unlocked one, a stray one, one under another key) and returns it.
  • func (f *Fake) List(_ context.Context, prefix string, versions bool, fn func(gcs.ObjectInfo) bool) error: List implements gcs.Store.
  • func (f *Fake) LoseResponseNext(method string, err error): LoseResponseNext makes the next call of method run and then return err: the effect happened and the caller does not know.
  • func (f *Fake) Names() []string: Names returns every object name, sorted.
  • func (f *Fake) Now() time.Time: Now is the fake's clock.
  • func (f *Fake) Overwrite(name string, body []byte): Overwrite replaces the live object with other bytes (a new generation; the old one becomes noncurrent), as a principal with storage.objects.create and storage.objects.delete can.
  • func (f *Fake) Read(_ context.Context, name string, generation, limit int64) ([]byte, gcs.ObjectInfo, error): Read implements gcs.Store.
  • func (f *Fake) ReleaseHold(name string, generation int64): ReleaseHold releases the temporary hold on a generation (what the separate hold-release identity does).
  • func (f *Fake) Report(name string, generation int64, fn func(*gcs.ObjectInfo)): Report changes what Cloud Storage reports for a generation (the live one when generation is 0), as a server that answers inconsistently would.
  • func (f *Fake) Stat(_ context.Context, name string, generation int64) (gcs.ObjectInfo, error): Stat implements gcs.Store.
  • func (f *Fake) Tamper(name string, generation int64, body []byte): Tamper rewrites the bytes of a generation in place, which Cloud Storage does not allow; it models damage or a forged listing that the Reader must catch by checking the bytes.
  • func (f *Fake) VanishOnRead(name string): VanishOnRead makes every read of name fail with ErrNoObject while its listing still shows it, as an object deleted at the end of its retention between a listing and a read.

Apache-2.0 licensed.