Bucket Event Notifications

Bucket event notifications allow you to receive real-time notifications when objects are created or deleted in your Storj buckets. This feature is S3-compatible and delivers notifications to Google Cloud Pub/Sub topics.

Overview

Bucket event notifications enable automated workflows when objects change in your buckets. Common use cases include:

• Media & Entertainment: Triggering transcoding pipelines when new video files are uploaded
• Quality Control: Starting automated QC workflows when assets land in a bucket
• Data Processing: Initiating ETL jobs when data files are created
• Backup Verification: Confirming successful uploads or tracking deletions

Supported Event Types

Delivery Guarantees

• At-least-once delivery: Events may occasionally be delivered more than once
• No ordering guarantee: Events may arrive out of order
• Low latency: Events are typically delivered within seconds of the object operation

Requirements

Before configuring bucket event notifications:

1. Project enabled for eventing: Contact Storj support to enable bucket eventing for your project
2. Storj-managed encryption: Your project must use Storj-managed encryption. This ensures object keys in notifications are readable. Projects with self-managed encryption are not supported.
3. Google Cloud Pub/Sub topic: You must create and own a Pub/Sub topic in your Google Cloud account

Setting Up Google Cloud Pub/Sub

Step 1: Create a Pub/Sub Topic

1. Go to the Google Cloud Console
2. Click Create Topic
3. Enter a topic ID (e.g., my-bucket-events)
4. Click Create

Note the fully-qualified topic name: projects/my-gcp-project-id/topics/my-topic-id

Step 2: Grant Permission to Storj

Storj needs permission to publish messages to your Pub/Sub topic. Grant the Pub/Sub Publisher role to the Storj bucket eventing service account for your satellite.

To grant access:

1. In the Google Cloud Console, navigate to your Pub/Sub topic
2. Click the Permissions tab
3. Click Add principal
4. In the "New principals" field, enter the Storj service account email for your satellite
5. Select the role Pub/Sub Publisher
6. Click Save

Step 3: Create a Subscription (Optional)

To receive the notifications, create a subscription for your topic:

1. In the Google Cloud Console, navigate to your Pub/Sub topic
2. Click Create Subscription
3. Choose subscription type:
   • Pull: Your application pulls messages from Pub/Sub
   • Push: Pub/Sub delivers messages to an HTTPS endpoint

For webhook delivery, use a Push subscription with your webhook URL.

Configuring Bucket Notifications

Filter Behavior

Filters allow you to receive notifications only for objects matching specific patterns:

• Filters are case-sensitive
• Prefix logs/ matches logs/2025.txt but NOT archive/logs/file.txt
• Suffix .jpg matches photo.jpg but NOT photo.JPG or image.jpg.bak
• When both prefix and suffix are specified, both must match (AND logic)
• Maximum filter length: 1024 characters

Event Message Format

Event notifications follow the S3 event message structure (version 2.1):

{
  "Records": [
    {
      "eventVersion": "2.1",
      "eventSource": "storj:s3",
      "eventTime": "2025-01-17T10:30:00.000Z",
      "eventName": "ObjectCreated:Put",
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "MyNotificationConfig",
        "bucket": {
          "name": "my-bucket",
          "ownerIdentity": {
            "principalId": "my-storj-project-id"
          },
          "arn": "arn:storj:s3:::my-bucket"
        },
        "object": {
          "key": "uploads/video.mp4",
          "size": 1048576,
          "versionId": "000000000000000190f2277f35af0b76",
          "sequencer": "1892E0DE46FBAE18"
        }
      }
    }
  ]
}

{
  "Records": [
    {
      "eventVersion": "2.1",
      "eventSource": "storj:s3",
      "eventTime": "2025-01-17T10:30:00.000Z",
      "eventName": "ObjectCreated:Put",
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "MyNotificationConfig",
        "bucket": {
          "name": "my-bucket",
          "ownerIdentity": {
            "principalId": "my-storj-project-id"
          },
          "arn": "arn:storj:s3:::my-bucket"
        },
        "object": {
          "key": "uploads/video.mp4",
          "size": 1048576,
          "versionId": "000000000000000190f2277f35af0b76",
          "sequencer": "1892E0DE46FBAE18"
        }
      }
    }
  ]
}

Message Fields

Test Event

When you first configure notifications, Storj sends a test event to verify the destination is reachable:

{
  "Service": "Storj S3",
  "Event": "s3:TestEvent",
  "Time": "2026-01-17T10:30:00.000Z",
  "Bucket": "my-bucket"
}

{
  "Service": "Storj S3",
  "Event": "s3:TestEvent",
  "Time": "2026-01-17T10:30:00.000Z",
  "Bucket": "my-bucket"
}

If the test event fails to publish, the configuration is rejected with an error message.

Setting Up a Webhook

To receive notifications at an HTTPS endpoint, configure a Pub/Sub Push subscription:

Step 1: Create a Push Subscription

1. In Google Cloud Console, go to Pub/Sub > Subscriptions
2. Click Create Subscription
3. Select your topic
4. Choose Push delivery type
5. Enter your webhook URL (must be HTTPS)
6. Configure authentication if needed
7. Click Create

Step 2: Verify Your Endpoint

Ensure your webhook endpoint:

• Accepts POST requests
• Returns a 2xx status code on success
• Handles JSON payloads
• Responds within 10 seconds (Pub/Sub timeout)

Example Webhook Handler (Node.js)

const express = require('express');
const app = express();

app.use(express.json());

app.post('/webhook', (req, res) => {
  const message = req.body.message;

  if (message && message.data) {
    // Decode base64 data
    const data = Buffer.from(message.data, 'base64').toString();
    const event = JSON.parse(data);

    // Log full event with nested objects expanded
    console.log('Received event:', JSON.stringify(event, null, 2));

    // Process the event
    for (const record of event.Records || []) {
      console.log(`Event: ${record.eventName}`);
      console.log(`Bucket: ${record.s3.bucket.name}`);
      console.log(`Object: ${record.s3.object.key}`);
    }
  }

  // Acknowledge receipt
  res.status(200).send('OK');
});

app.listen(3000, () => {
  console.log('Webhook server listening on port 3000');
});

const express = require('express');
const app = express();

app.use(express.json());

app.post('/webhook', (req, res) => {
  const message = req.body.message;

  if (message && message.data) {
    // Decode base64 data
    const data = Buffer.from(message.data, 'base64').toString();
    const event = JSON.parse(data);

    // Log full event with nested objects expanded
    console.log('Received event:', JSON.stringify(event, null, 2));

    // Process the event
    for (const record of event.Records || []) {
      console.log(`Event: ${record.eventName}`);
      console.log(`Bucket: ${record.s3.bucket.name}`);
      console.log(`Object: ${record.s3.object.key}`);
    }
  }

  // Acknowledge receipt
  res.status(200).send('OK');
});

app.listen(3000, () => {
  console.log('Webhook server listening on port 3000');
});

Limitations

Troubleshooting

Configuration Fails with Permission Error

Ensure the Storj service account has the Pub/Sub Publisher role on your topic. Double-check:

• The service account email is correct for your satellite
• The role is granted at the topic level (not the subscription level)

IAM Policy Update Failed (Domain Restricted Sharing)

If you see an error like:

IAM policy update failed The 'Domain Restricted Sharing' organization policy (constraints/iam.allowedPolicyMemberDomains) is enforced. Only principals in allowed domains can be added as principals in the policy.

This occurs when your Google Cloud organization has the Domain Restricted Sharing policy enabled, which prevents adding external service accounts (like Storj's) to your IAM policies.

Solution: You need to add Storj's organization to the allowed domains in your organization policy:

1. In the Google Cloud Console, ensure you have the Organization selected (not a project)
2. Navigate to IAM & Admin > Organization Policies
3. Find and click on Domain restricted sharing (iam.allowedPolicyMemberDomains)
4. Click Manage policy
5. Under Rules, expand the existing Allow rule
6. Click Add value to add a new custom value
7. Add principalSet://iam.googleapis.com/organizations/693710676402 (Storj's organization ID)
8. Click Done, then Set policy

After updating the organization policy, return to your Pub/Sub topic and grant the Pub/Sub Publisher role to the Storj service account.

Not Receiving Events

1. Check your bucket eventing configuration: Use GetBucketNotificationConfiguration to verify the configuration is set correctly
2. Check event types: Ensure the event type you expect is in your configuration
3. Check filters: Verify your prefix/suffix filters match the objects you're uploading
4. Check your subscription: Ensure you have an active subscription to the Pub/Sub topic
5. Check Pub/Sub logs: Review Cloud Logging for any delivery errors

Delayed Events

Events are typically delivered within seconds, but may be delayed during:

• High system load
• Network issues between Storj and Google Cloud

Duplicate Events

Due to at-least-once delivery, you may occasionally receive duplicate events. Design your application to handle duplicates idempotently using the sequencer field.

S3 Compatibility

API Methods

Destination Types

Event Types

Filters

Other Features

Cost Considerations

Bucket eventing incurs costs on Google Cloud for Pub/Sub usage, including throughput and message storage. See the Google Cloud Pub/Sub pricing page for current rates.

Storj does not currently charge for the bucket eventing feature itself.