AI observability creates a useful but uncomfortable security tradeoff.
Security and engineering teams need enough evidence to answer questions such as:
Which identity invoked the model?
Which model and operation were used?
Did the request fail?
How many tokens were consumed?
Did a prompt or response violate policy?
What information was supplied to the model?
The most detailed logs may also contain prompts, responses, documents, images, and metadata. That means enabling every available logging option can create a sensitive-data repository that did not previously exist.
While reviewing Amazon Bedrock invocation-logging settings, I was reminded that the goal is not simply to turn logging on. The goal is to collect justified evidence, protect it appropriately, and verify that the complete process works.
Understand what the feature captures
Amazon Bedrock model invocation logging can collect request data, response data, and invocation metadata. Supported destinations include CloudWatch Logs and Amazon S3, and the configuration applies within an AWS account and Region.
AWS also states that invocation logging is disabled by default. Supported runtime operations include Converse, ConverseStream, InvokeModel, and InvokeModelWithResponseStream. Calls through other endpoints may not be captured. Amazon Bedrock invocation-logging documentation. Amazon Bedrock
That introduces an important validation question:
Does the logging configuration cover the actual API path the application uses?
A configured destination does not prove that every intended invocation is covered.
Classify the content before selecting modalities
Bedrock allows logging for text, images, embeddings, and video.
I would not enable each modality automatically. I would first document:
What data the application is permitted to send.
Whether prompts may contain personal, confidential, regulated, or proprietary information.
Whether full input and output content is necessary.
Which team needs the content.
How long that content remains useful.
Whether metadata and metrics can satisfy some requirements without full content.
Large inputs, documents, images, and larger payloads can be written to S3. That destination should therefore be treated as a potentially sensitive application-data store—not merely an operational log bucket.
Separate metrics from content
Different evidence supports different use cases.
For operational monitoring, invocation counts, latency, token consumption, throttling, and error rates may be enough.
For content-safety investigations, prompt and response content may be necessary.
For cost allocation, an internal workload or team identifier may be sufficient.
AWS specifically warns against placing personal information, credentials, secrets, or regulated data in request metadata or session tags because those values can surface in invocation logs or cost records. Amazon Bedrock cost-attribution guidance. Amazon Bedrock
My default is to collect the minimum data needed for the documented purpose.
Design least-privilege roles
The identities that configure logging, publish records, administer storage, and investigate events do not need identical access.
I would separate:
Logging-configuration administration.
Bedrock delivery permissions.
Storage and encryption administration.
Read-only operational analysis.
Approved access to full prompt and response content.
Deletion or retention administration.
An analyst who needs token counts and failure metadata may not need to read full user prompts. A storage administrator may not need permission to invoke a model.
This separation supports least privilege and improves accountability.
Protect the destinations
For CloudWatch Logs and S3, I would evaluate:
Encryption requirements.
Destination policies and service-role trust.
Public-access protections.
Cross-account access.
Retention and lifecycle rules.
Administrative logging.
Deletion protections where justified.
Detection for delivery failures or unexpected configuration changes.
AWS requires Bedrock logging destinations to be in the same account and Region as the configuration. That affects designs that otherwise centralize security data directly across accounts. Amazon Bedrock
If centralized analysis is required, I would design a separate, controlled aggregation process rather than assuming the initial destination can be shared across every account and Region.
Validate the full evidence path
I would use an approved test invocation containing synthetic, non-sensitive data.
The validation should confirm:
The application uses an operation covered by the logging feature.
The invocation creates the expected record.
Metadata fields identify the expected model, identity, Region, and operation.
Selected content appears only in the intended destination.
Large or binary content follows the expected S3 path.
The investigator role can retrieve the required evidence.
An unrelated role cannot retrieve restricted content.
Retention settings match the documented policy.
The result should be recorded as evidence. A configuration screen displaying selected destinations is not equivalent to confirmed delivery.
Monitor the monitoring
Logging can fail after deployment because of policy changes, deleted destinations, altered encryption permissions, or application changes.
Defense in depth means monitoring:
Configuration drift.
Delivery failures.
Unexpected gaps in invocation volume.
Unauthorized access to the records.
Retention-policy changes.
Applications moving to unsupported invocation paths.
Zero Trust means verifying that evidence continues to arrive rather than trusting the initial configuration.
Treat logs as part of AI governance
NIST’s Generative AI Profile applies risk management across the AI lifecycle and helps organizations align AI controls with their goals, obligations, and priorities. Logging should support that broader governance process rather than exist as an isolated technical feature. NIST Generative AI Profile. NIST
For each AI workload, I would document:
Business owner.
Approved purpose and data.
Model and access path.
Logging purpose.
Content captured.
Authorized investigators.
Retention period.
Incident-response path.
Review cadence.
Good AI logging improves accountability without unnecessarily recreating sensitive data in another location.
Explore my related Security Engineering Case Studies.