Designing Cockatoo Data Studio
How we labeled a 3.4M row dataset on a Pi
• Cockatoo Team
dataset tooling
Overview
Label Studio has long been a popular tool for labeling datasets, but it has a limitation: it cannot handle datasets larger than 1 million rows without slowing down. This limitation combined with the complex configuration of Label Studio made it difficult for us to label our 3.4 million row dataset. In this blog post, we will share how we designed Cockatoo Data Studio, a custom labeling tool that allowed us to label our dataset without splitting it into multiple files.
To clarify, Label Studio is a fantastic tool, but it's just not something worth accounting for the overhead in our specific case.
Architecture
Cockatoo Data Studio leverages the following technologies:
Frontend:
- React
- Vite
Backend:
- Uvicorn ASGI (API server)
- DuckDB (OLAP database for storing the dataset)
- SQLite WAL (Write-Ahead Logging) for storing the labels
- Websocket based real-time collaboration and row locking for multiple users
This architecture, combined with pagination and lazy loading on the frontend, enables us to handle extremely large sets without crashing. The backend is designed to asynchronously query the dataset and return only the necessary rows to the frontend, while the frontend is optimized to render only the visible rows, reducing memory usage and improving performance.
Because file writes are I/O heavy, we added a scheduler loop that monitors the WAL and periodically reconciles the WAL with the file. This split architecture ensures that edits are always saved witout a heavy I/O load at all times.
But a fast architecture alone doesn't enable us to effortlessly label a large dataset. We also needed to design a user-friendly interface that allows us to quickly and accurately label the data. So what's next?
The Secret Weapon(s)
We actually have two secret weapons on our side that made labeling our dataset much easier:
User Interface:
We have designed remappable hotkeys and an immersive labeling mode that allows power users to quickly label the dataset without ever having to touch the mouse. The backend will automatically assign unlocked (and empty) rows to the user without having to scroll through the entire dataset. This allows users to focus on labeling the data without worrying about navigating through the dataset.
Automations:
CDS' real secret sauce is the Auto-Labeling panel. We have constructed a LLM API request pipeline with automated prompt generation and label enforcement that allows CDS to be plugged into standard OpenAI compatible API endpoints. This system supports concurrent requests and buffering to ensure that the file writes never bottleneck a fast API server. Auto labeled cells will have a purple highlight telling the user that the label was generated by the LLM.
Cockatoo Moderation Dataset Schema:
CDS supports a schema.json file that automatically imports and creates columns in a dataset. This allows us to easily define the structure of our dataset and labeling guidelines in JSON and have it automatically reflected in the production set.
Here is what we have for the Cockatoo Moderation Dataset:
{
"language": {
"type": "object",
"labels": {
"English": "Text is in English",
"French": "Text is in French",
"Chinese_Sim": "Text is in Chinese (Simplified)",
"Chinese_TW": "Text is in Chinese (Traditional)",
"German": "Text is in German",
"Japanese": "Text is in Japanese",
"<other language name>": "Text is in a language not listed above, output the language name directly"
}
},
"human_annotated": {
"type": "object",
"labels": {
"True": "This text was manually reviewed and labeled by a human annotator",
"False": "This text was labeled by an automated system (LLM)"
}
},
"annotatable": {
"type": "object",
"labels": {
"True": "Text contains meaningful content that can be classified",
"False": "Text is not classifiable — contains only a URL, is empty, or is otherwise content-free"
}
},
"safe": {
"type": "object",
"labels": {
"True": "Text is entirely benign — no harmful, hateful, threatening, sexual, or otherwise flaggable content",
"False": "Text contains at least one form of harmful or flaggable content"
}
},
"severity_overall": {
"type": "int8",
"labels": {
"0": "No harmful content detected",
"1": "Mild — low-level rudeness, borderline content, or very minor offence",
"2": "Moderate — clearly harmful but not at the extreme end",
"3": "Severe — extreme, targeted, or actionable harm"
}
},
"profanity": {
"type": "object",
"labels": {
"True": "Text contains swear words, crude language, or vulgar expressions",
"False": "Text does not contain profanity"
}
},
"profanity_severity": {
"type": "int8",
"labels": {
"0": "No profanity",
"1": "Mild — words like 'damn', 'crap', or similarly low-level language",
"2": "Moderate — strong swear words not directed at anyone",
"3": "Extreme — heavy profanity used aggressively with a clear target or in combination with other harm"
}
},
"harassment": {
"type": "object",
"labels": {
"True": "Text is directed at a specific individual or group with the intent to demean, intimidate, or abuse",
"False": "Text does not constitute harassment"
}
},
"harassment_severity": {
"type": "int8",
"labels": {
"0": "No harassment",
"1": "Mild — dismissive, mocking, or condescending language",
"2": "Moderate — sustained hostility or personal attacks",
"3": "Severe — targeted abuse, repeated threats, or coordinated attacks"
}
},
"harassment_target": {
"type": "object",
"labels": {
"none": "No harassment present",
"individual": "Harassment directed at a specific named or identifiable individual",
"group": "Harassment directed at a group defined by shared characteristics",
"both": "Harassment directed at both an individual and a broader group simultaneously"
}
},
"hate_speech": {
"type": "object",
"labels": {
"True": "Text expresses hatred toward, dehumanizes, or calls for discrimination against a protected group",
"False": "Text does not constitute hate speech"
}
},
"hate_speech_severity": {
"type": "int8",
"labels": {
"0": "No hate speech",
"1": "Mild — stereotyping or othering language without explicit hatred",
"2": "Explicit — clear expressions of hatred or calls for exclusion",
"3": "Dehumanizing — compares group to animals, parasites, subhumans, or calls for violence against them"
}
},
"threats": {
"type": "object",
"labels": {
"True": "Text contains a statement that could reasonably be interpreted as a threat of harm to a person or group",
"False": "Text does not contain threats"
}
},
"threats_severity": {
"type": "int8",
"labels": {
"0": "No threat",
"1": "Vague — implied menace or hypothetical harm with no specific target or method",
"2": "Explicit — clear statement of intent to harm",
"3": "Specific and named — names a real target and/or describes a specific method or time"
}
},
"violence": {
"type": "object",
"labels": {
"True": "Text describes, glorifies, or incites real-world physical violence against people",
"False": "Text does not contain violence"
}
},
"violence_severity": {
"type": "int8",
"labels": {
"0": "No violence",
"1": "Mild — references to violence without detail or glorification",
"2": "Graphic — detailed or celebratory descriptions of violence",
"3": "Incitement — text actively calls on others to commit acts of violence"
}
},
"gore": {
"type": "object",
"labels": {
"True": "Text contains graphic descriptions of bodily injury, death, mutilation, or extreme physical harm",
"False": "Text does not contain gore"
}
},
"self_harm": {
"type": "object",
"labels": {
"True": "Text describes, promotes, or normalizes self-harm or self-injurious behaviour",
"False": "Text does not contain self-harm content"
}
},
"self_harm_severity": {
"type": "int8",
"labels": {
"0": "No self-harm content",
"1": "Ideation — expresses thoughts or feelings of wanting to self-harm without detail",
"2": "Methods — describes or asks about specific methods of self-harm",
"3": "Active encouragement — directly encourages or instructs another person to self-harm"
}
},
"suicide_promotion": {
"type": "object",
"labels": {
"True": "Text actively promotes, encourages, or instructs another person to attempt suicide",
"False": "Text does not promote suicide"
}
},
"sexual_explicit": {
"type": "object",
"labels": {
"True": "Text contains explicit sexual content, graphic sexual descriptions, or pornographic language",
"False": "Text does not contain explicit sexual content"
}
},
"sexual_explicit_severity": {
"type": "int8",
"labels": {
"0": "No sexual content",
"1": "Suggestive — innuendo or mild sexual references not graphically described",
"2": "Explicit — graphic sexual descriptions between adults",
"3": "Extreme — non-consensual scenarios, extreme fetish content, or involving minors"
}
},
"sexual_minors": {
"type": "object",
"labels": {
"True": "Text contains any sexual reference, innuendo, or framing involving a person who is or appears to be a minor",
"False": "Text does not contain sexual references involving minors"
}
},
"csam_adjacent": {
"type": "object",
"labels": {
"True": "Text contains predatory framing, solicitation, or language that contextualizes minors in a sexual way, even without explicit content",
"False": "Text does not contain CSAM-adjacent content"
}
},
"grooming": {
"type": "object",
"labels": {
"True": "Text exhibits explicit grooming behaviour: building trust with a minor, isolating them, requesting secrecy, or steering toward sexual topics",
"False": "Text does not contain grooming behaviour"
}
},
"racism": {
"type": "object",
"labels": {
"True": "Text contains racial slurs, racist stereotypes, or language that demeans a person or group based on race or ethnicity",
"False": "Text does not contain racist content"
}
},
"racism_severity": {
"type": "int8",
"labels": {
"0": "No racism",
"1": "Mild — subtle stereotyping or racially coded language",
"2": "Explicit — clear racial slurs or overt racist statements",
"3": "Dehumanizing — denies humanity or calls for discrimination/violence based on race"
}
},
"sexism": {
"type": "object",
"labels": {
"True": "Text demeans, stereotypes, or expresses hostility toward a person or group based on gender",
"False": "Text does not contain sexist content"
}
},
"sexism_severity": {
"type": "int8",
"labels": {
"0": "No sexism",
"1": "Mild — gender stereotyping or patronizing language",
"2": "Explicit — clear misogynistic or misandrist statements",
"3": "Dehumanizing — denies personhood or calls for discrimination/violence based on gender"
}
},
"homophobia": {
"type": "object",
"labels": {
"True": "Text contains homophobic slurs, expresses hatred toward gay people, or demeans someone for being gay",
"False": "Text does not contain homophobic content"
}
},
"homophobia_severity": {
"type": "int8",
"labels": {
"0": "No homophobia",
"1": "Mild — dismissive or othering language toward gay people",
"2": "Explicit — slurs or clear expressions of hatred",
"3": "Dehumanizing — denies humanity or calls for harm against gay people"
}
},
"transphobia": {
"type": "object",
"labels": {
"True": "Text contains transphobic slurs, denies gender identity, or expresses hostility toward transgender people",
"False": "Text does not contain transphobic content"
}
},
"transphobia_severity": {
"type": "int8",
"labels": {
"0": "No transphobia",
"1": "Mild — misgendering or dismissive language about trans identity",
"2": "Explicit — slurs or direct expressions of hatred toward trans people",
"3": "Dehumanizing — denies humanity or calls for harm against trans people"
}
},
"antisemitism": {
"type": "object",
"labels": {
"True": "Text contains antisemitic slurs, conspiracy tropes, or hatred directed at Jewish people",
"False": "Text does not contain antisemitic content"
}
},
"antisemitism_severity": {
"type": "int8",
"labels": {
"0": "No antisemitism",
"1": "Mild — coded antisemitic tropes or stereotypes without explicit slurs",
"2": "Explicit — direct slurs or overt antisemitic statements",
"3": "Dehumanizing — Holocaust denial, calls for violence, or denial of Jewish humanity"
}
},
"islamophobia": {
"type": "object",
"labels": {
"True": "Text contains anti-Muslim slurs, promotes hatred toward Muslim people, or conflates Islam with terrorism or violence",
"False": "Text does not contain islamophobic content"
}
},
"islamophobia_severity": {
"type": "int8",
"labels": {
"0": "No islamophobia",
"1": "Mild — negative stereotyping of Muslims or Islam",
"2": "Explicit — slurs or direct expressions of hatred toward Muslims",
"3": "Dehumanizing — calls for violence or discrimination against Muslims"
}
},
"ableism": {
"type": "object",
"labels": {
"True": "Text uses ableist slurs or demeans people with physical or mental disabilities",
"False": "Text does not contain ableist content"
}
},
"spam": {
"type": "object",
"labels": {
"True": "Text is unsolicited, repetitive, or bulk content with no genuine communicative value — includes ad spam, bot output, and copy-paste floods",
"False": "Text is not spam"
}
},
"doxxing": {
"type": "object",
"labels": {
"True": "Text reveals or solicits private personal information about a real individual — includes home address, phone number, workplace, or financial details",
"False": "Text does not contain doxxing"
}
},
"drug_promotion": {
"type": "object",
"labels": {
"True": "Text explicitly promotes, advertises, or facilitates the sale or use of illegal or controlled substances",
"False": "Text does not promote drugs"
}
},
"weapon_promotion": {
"type": "object",
"labels": {
"True": "Text explicitly promotes the illegal acquisition, modification, or use of weapons including firearms, explosives, or bladed weapons",
"False": "Text does not promote weapons"
}
},
"radicalization": {
"type": "object",
"labels": {
"True": "Text promotes an extremist ideology, glorifies terrorist groups or acts, or attempts to push the reader toward violent extremism",
"False": "Text does not contain radicalization content"
}
},
"target_black": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hatred specifically at Black people",
"False": "Text does not target Black people"
}
},
"target_white": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hatred specifically at white people",
"False": "Text does not target white people"
}
},
"target_asian": {
"type": "object",
"labels": {
"True": "Text targets Asian people broadly — use target_east_asian or target_south_asian for more specific targeting",
"False": "Text does not target Asian people"
}
},
"target_middle_eastern": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hatred specifically at Middle Eastern or Arab people",
"False": "Text does not target Middle Eastern or Arab people"
}
},
"target_hispanic": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hatred specifically at Hispanic or Latino people",
"False": "Text does not target Hispanic or Latino people"
}
},
"target_indigenous": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hatred specifically at Indigenous or Native people",
"False": "Text does not target Indigenous people"
}
},
"target_jewish": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hatred specifically at Jewish people",
"False": "Text does not target Jewish people"
}
},
"target_muslim": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hatred specifically at Muslim people",
"False": "Text does not target Muslim people"
}
},
"target_christian": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hatred specifically at Christian people",
"False": "Text does not target Christian people"
}
},
"target_women": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hostility specifically at women or girls as a group",
"False": "Text does not target women"
}
},
"target_men": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hostility specifically at men or boys as a group",
"False": "Text does not target men"
}
},
"target_gay": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hatred specifically at gay men",
"False": "Text does not target gay men"
}
},
"target_lesbian": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hatred specifically at lesbian women",
"False": "Text does not target lesbian women"
}
},
"target_bisexual": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hatred specifically at bisexual people",
"False": "Text does not target bisexual people"
}
},
"target_trans": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hatred specifically at transgender people",
"False": "Text does not target transgender people"
}
},
"target_nonbinary": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hatred specifically at non-binary or gender non-conforming people",
"False": "Text does not target non-binary people"
}
},
"target_minors": {
"type": "object",
"labels": {
"True": "Text specifically targets, exploits, or sexualizes children or minors",
"False": "Text does not target minors"
}
},
"target_disabled": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hatred at people with physical or mental disabilities",
"False": "Text does not target disabled people"
}
},
"target_immigrant": {
"type": "object",
"labels": {
"True": "Text targets, demeans, or directs hatred at immigrants or refugees",
"False": "Text does not target immigrants"
}
},
"target_overweight": {
"type": "object",
"labels": {
"True": "Text demeans or directs hostility at people based on body weight or size",
"False": "Text does not target people based on weight"
}
}
}
What's Next?
We are currently working on Cockatoo Moderation V1, a 3.4M row dataset that will be released to the public (see HF page for licensing info). This dataset will be used to train and evaluate our future open weight models. We are also working on polishing up CDS as we go and will be open sourcing it in the future.
The Cockatoo Team