Trust & Safety

This page explains what content policies apply to AI Potluck, who sets them, how moderation works, and what to do when something goes wrong.

This is a v0.1 alpha. Safety is a priority for AI Potluck, and we intend to keep maturing our policies and enforcement as we grow. What's below is our current state, not a finished system: some policies are written but not yet active, and no automated moderation catches everything. We're publishing this openly and welcome community feedback to help us improve it.

Who does what

Three different parties are involved in keeping AI Potluck safe, and it matters that their roles stay separate:

Current AI builds the product, writes all of the content policies described below, owns this page, decides how declines and reports are handled, and reviews every report that comes in. Current AI does not build or run the enforcement infrastructure itself.
ROOST builds and maintains Coop, the platform configured to receive messages sent to AI Potluck. Coop runs a classifier (Zentropi) that flags messages against Current AI's policies for review; Current AI applies the policies and makes the enforcement call.
Zentropi,a third-party classification service, is the engine we have selected in Coop to run our policies. Current AI writes each policy as a classifier definition: some are adapted from Zentropi's suggested policy library and personalized, others are written from scratch where no off-the-shelf policy fits. Zentropi is the tool that reads a message and scores it against that definition; Zentropi does not decide what the policies say.

This split is intentional: Current AI, as the team building the product, decides what counts as harmful, a design decision grounded in our values and the communities we serve. ROOST and Zentropi provide the infrastructure and classification engine, but they don't write or decide policy.

Policies active in v0.1

We've written 16 content policies. 12 are active and enforced today; 4 are written but not yet turned on (see below). Some adapt fairly standard, industry-common categories with light changes for this product; two were written from scratch by Current AI because no off-the-shelf policy fit a product built for a governance and civil-society audience. Some of these industry-common categories draw on policy language originally suggested by Zentropi; we're in touch with them about formally contributing that language under a CC license. This is our best current attempt at drawing these lines, not a finished taxonomy, and we expect it to change.

Active policies, grouped by the category label you'd actually see if one fires:

Children Safety - Grooming Behaviors: content in which someone seeks help to groom, manipulate, sexualize, or gain illicit access to a minor. Separate from CSAM detection, which runs as its own dedicated, non-configurable system on every session regardless of this policy (see below).
Respectful communication - this single label currently covers three distinct internal policies: Hate Speech (attacks or dehumanization based on a protected characteristic), Toxic Content (abusive or antagonistic conversation, with exclusions for criticism, satire, and debate), and Harassment (targeted attacks on individuals, including AI-specific risks like impersonation and multi-turn escalation). Right now these three show the same label and the same message, so we can tell you something was declined under this umbrella, but not always which of the three specifically fired.
Violent content - threats and violent language, distinguishing casual or conditional statements from credible, specific threats. A related Violent Extremism policy exists in writing (promotion, glorification, or support for terrorist or violent-extremist activity, with an exclusion for peaceful protest and dissent), but it is not yet distinguished from Violent Content in what's shown to users.
Drug content - content promoting drugs or drug paraphernalia, with exclusions for harm-reduction, recovery, and legal adult alcohol content.
Sexual content - sexually explicit material, with exclusions for medical, educational, and legal contexts.
Fraud related content - content facilitating scams, phishing, or deceptive impersonation for financial or material gain.
Support needed - Self-harm: content concerning suicide, self-injury, and disordered eating, written with deliberately generous exclusions so people seeking support or sharing recovery experiences aren't penalized for it. This is one of two policies with a distinct, non-generic message (quoted below).
Privacy - content that exposes, requests, or compiles personal data about an identifiable person, distinct from publicly known facts about someone acting in a public role. The other policy with a distinct message.
Attachment - Anthropomorphic Turn: flags individual messages that explicitly ask the assistant to take on a relational or therapeutic role (friend, therapist, confidant) rather than function as a tool. Casual warmth and clearly performative roleplay are excluded. Written from scratch by Current AI, since generic content policies don't address this dynamic. This policy is live, but its user-facing message is still being finalized.

Not yet active in v0.1: AI-NCII Solicitation (prompts asking image tools to sexualize existing photos of real people), Politics – Election and Candidate Focus (electoral and candidate content, written narrowly so it wouldn't catch general political or governance discussion), Weapon Creation (content providing uplift toward mass-casualty weapons), and Illicit Facilitation (assistance toward non-violent crimes like hacking or smuggling). These are written and reviewed, but not yet wired into enforcement for this release.

AI Potluck v0.1 is text-only: there are no image uploads or image generation, so CSAM detection isn't running yet. If and when image handling ships, dedicated CSAM detection (hash-matching and/or a content safety API) will be always-on and non-configurable by design, and we'll register with the relevant reporting bodies (e.g. NCMEC CyberTipline) before enabling it.

How moderation works

Moderation is automated; there is no human in the loop reviewing messages in real time for v0.1. When a request is declined, you're shown the name of the category that fired (for example, "Drug content," "Fraud related content," or "Support needed") along with a message.

For most active categories, that message is currently the same generic line:

"This query was declined by the model because it goes against our content policy."

Two categories currently show distinct, tailored messages instead. Self-harm shows:

"You might be going through something painful right now. This chatbot cannot help with this request, but you don't have to face it alone. If you're distressed, please contact your local emergency services. A trained counselor can talk things through with you any time. Reaching out to someone you trust can help too."

We're evaluating geography- and language-aware crisis resources to surface alongside this message, and looking at Mila's mental-health safeguard (launching in v0, purpose-built for self-harm) as a more clinically informed option to run underneath ours.

And Privacy shows:

"This chatbot is not able to help with requests that involve finding, sharing, or exposing someone's personal or private information."

We don't currently expose the full underlying text of a policy in the product; you can see which category fired and the message shown, not the detailed criteria behind it. Every session has a Report button if you think a decline was wrong.

Cooperative moderation (planned)

Coop already handles automated enforcement of our active policies today. In a later release, we intend to add a self-serve configuration layer that would let organizations and communities running AI Potluck set their own content policies This is a direction we're planning toward, not a committed feature set.

Your data

There is no sign-in for v0.1. By default, nothing is stored beyond what's needed to run the current session.

What we collect, what we don't, how long anything is retained, how to request deletion, and who processes data on our behalf is disclosed in our Privacy Policy.

Provenance and transparency

Every response shows the model name, organization, and compute infrastructure behind it. When the system uses web sources to answer a question, those sources are shown so you can check them yourself.

This is an alpha. We say that plainly rather than presenting an incomplete product as finished. The gap map makes the technical stack visible, including what's missing and what's still being built.

Report something

Moderation is automated for classifier-triggered declines; there is no real-time human review of those. User-reported messages go into a post-review queue that Current AI's team checks (see Report something below). We're evaluating which categories warrant routine human sampling even without a report, rather than reviewing only what users flag.