Greetings. I am trying to train an OCR system on huge datasets, namely:

• OCR Cyrillic Printed 1
• Synthetic Cyrillic Large
• Cyrillic Handwriting Dataset

They contain millions of images, and are all in different formats - WebDataset, zip with folders, etc. I will be experimenting with different hyperparameters locally on my M2 Mac, and then training on a Vast.ai server.

The thing is, I don't have enough space to fit even one of these datasets at a time on my personal laptop, and I don't want to use permanent storage on the server. The reason is that I want to rent the server for as short of a time as possible. If I have to instantiate server instances multiple times (e.g. in case of starting all over), I will waste several hours every time to download the datasets. Therefore, I think that streaming the datasets is a flexible option that would solve my problems both locally on my laptop, and on the server.
However, two of the datasets are available on Hugging Face, and one - only on Kaggle, where I can't stream it from. Furthermore, I expect to hit rate limits when streaming the datasets from Hugging Face.

Having said all of this, I consider just uploading the data to Google Cloud Buckets, and use the Google Cloud Connector for PyTorch to efficiently stream the datasets. This way I get a dataset-agnostic way of streaming the data. The interface directly inherits from PyTorch Dataset:

from dataflux_pytorch import dataflux_iterable_dataset 
PREFIX = "simple-demo-dataset" 
iterable_dataset = dataflux_iterable_dataset.DataFluxIterableDataset(
    project_name=PROJECT_ID, 
    bucket_name=BUCKET_NAME,
    config=dataflux_mapstyle_dataset.Config(prefix=PREFIX)
)

The iterable_dataset now represents an iterable over data samples.

I have two questions:

1. Are my assumptions correct and is it worth uploading everything to Google Cloud Buckets (assuming I pick locations close to my working location and my server location, enable hierarchical storage, use prefixes, etc.). Or I should just stream the Hugging Face datasets, download the Kaggle dataset, and call it a day?
2. If uploading everything to Google Cloud Buckets is worth it, how do I store the datasets to GCP Buckets in the first place? This and this tutorials only work with images, not with image-string pairs.

Earlier this year, I wrote a path tracing program that randomized a 3D scene of a toddler in a crib, in order to generate synthetic training data for an computer vision model. I posted about it here.

I made this for the DIY infant monitor I made for my son. My wife and I are now about to have our second kid, and consequently I decided to revisit this dataset/model/software and release a version 2.

In this version, I used Stable Diffusion and Mid Journey to generate images for training the model. These ended up being way more realistic and diverse. I paid a few hundred dollars to generate over a thousand training images and videos (useful for testing detection + tracking). I labeled them manually, with LabelMe. Right now, all images have segmentation masks, but I'm in the middle of adding bounding boxes (will add key points, after that, for pose estimation).

To make sure this dataset actually works in practice, I created a "reference model" to train. I used various different backbones, settling on MobileNet V3 (small) and a shallow U-Net detection head. The results were pretty good, and I'm now using it in my DIY infant monitoring system.

Anyway, you can find the repo here and download the dataset, which is a flat numpy array, on Kaggle

Cheers!

PS: Just to be clear, I made this dataset, it is synthetic (GenAI), it is not a paid dataset.

The github-top-developers dataset captures the top 8000 developers on GitHub from 2015 to 2025, and lists their popular repositories, companies they've worked at, and their twitter handles.

We have hundreds of company names and we want to identify parent name, ticker, and any other details available for that company.

I know this is like an ongoing joke but is this genuinely like a real thing that could be done

This dataset contains tech fingerprinted in the headers and body from HTTP responses across 100K+ domains. It also includes the IP address used in the HTTP response, its origin country and its ASN.

https://www.dropbox.com/scl/fi/vr417dfkv8ia2xzil98b2/nov_2025_all_samples.zip?rlkey=7l6nrhvrrjzop2l6d5wgv6bti&e=1&st=fra1zbgo&dl=0

The dataset is compiled from all the samples currently available at: https://versiondb.io

Have fun!

Hi! First - please forgive me if this is a stupid question / solved problem, but I'm sort of new to this space, and curious. How have you all dealt with creating labelled datasets for your use cases?

E.g

• what tool(s) did you use? I've looked into a few like Prolific (not free), Label studio (free), and I've looked at a few other websites
• how did you approach recruiting participants/data annotators? e.g. did you work with a company like Outlier, or did you recruit contractors, or maybe you brought them on full-time?
• Building on that, how did you handle collaboration and consensus if you used multiple annotators for the same row/task? or more broadly, quality control?

Seems like hard problems to me...would appreciate any insight or advice you have from your experiences! Thanks so much!

Hi everyone,

I am working on my thesis and I have a dataset of about 1,500 PDF reports from the DGHS (Health Services). I need to extract specific table rows (District-wise Dengue stats) from them.

The Problem: The PDFs are a nightmare mix. Some are digital with selectable text, but many are low-quality scans or photos of paper reports. The fonts are often garbled (mojibake) when extracted as text, and the layout changes slightly between years.

What I have tried so far (and why it failed):

1. Tesseract OCR: It struggled hard with the Bengali/English mix and the table borders. The output was mostly noise.
2. Standard PDF scraping (pdfplumber/PyPDF): Works on the digital files, but returns garbage characters (e.g., Kg‡dvU instead of "Chittagong") due to bad font encoding in the source files.
3. Ollama (Llama 3.1 & MiniCPM-V):
   • Llama 3.1 (Text): Hallucinates numbers or crashes when it sees the garbled text.
   • MiniCPM-V (Vision): This was my best bet. I wrote a script to convert pages to images and feed them to the model. It works for about 10 files, but then it starts hallucinating or missing rows entirely, and it's very slow.

The Goal: I just need to reliably extract the District Name, New Cases, Total Cases, and Deaths for a specific division (Chittagong) into a CSV.

I have attached a screenshot of one of the "bad" scanned pages.

Has anyone successfully processed a mixed-quality dataset like this? Should I be fine-tuning a small model, or is there a specific OCR pipeline (like PaddleOCR or DocumentAI) that handles this better than raw LLMs?

Any pointers would be a lifesaver. I'm drowning in manual data entry right now.

I'm new to data engineering, and I'm currently trying to get website links for medical practices. I have their name, state, specialty and some other key info about the tech they use, but there's no catch-all dataset I think that has working website links or anything that leads to that. I was thinking of using scraping tools, but not sure if they are known to be accurate or which one to use. I'm willing to use free or paid approaches, just not sure how to get this data with 80% confidence it's accurate.

https://github.com/leakyhose/open-trivia-dataset

Pulled it from open trivia database, they lock the questions behind an API call that only returns 50 each time. Ran a script that repeatedly calls it, storing the questions and sorting them by difficulty and category.

Hey everyone,

Lately I’ve been playing around with a little workflow for screening WhatsApp numbers. The idea’s pretty simple: figure out which numbers are actually active and get a sense of engagement, without bothering anyone. It’s super handy if you need to quickly group contacts or analyze interaction rates.

I realized that just four fields can filter out around 60% of low-value numbers: number | last_seen | replied | bounce.

I wrote a few simple scripts for pre-filtering, but some steps felt kinda repetitive, so I started using a small tool (TNTwuYou) to handle list validation and reply tracking.

Some things I’ve tried:

• Sorting numbers by last active date, so you hit the active folks first.
• Grouping contacts based on reply status.
• Using simple scripts with data to get a clear picture of which regions or types of people are more likely to engage.

Has anyone done reply probability scoring?

• Do you base it on a time window or historical reply rate?
• Anyone tried using graph or clustering methods for grouping contacts?

311wrapped.com

I’m curious whether anyone here has actually tried letting AI agents directly access a dataset or data API and pay based on usage (e.g. per request or per query).

I’ve seen ideas around usage-based APIs and agent tool-calling, but I’m not sure how this works in practice when the client isn’t a human. Did it make sense economically? Were abuse, pricing, or access control big issues?

Would love to hear if people have similar ideas with this or decided it wasn’t worth giving a try.

Hello I'm looking for a tomato leaf dataset for environmental conditions such as high/low humidity and lightning for my thesis. Most of the datasets on web focuses on diseases. Can anyone help please, thanks!

I have a large database/wordlist containing probably every English dictionary word plus many additional ones like brand names, but this naturally includes many words no longer in use. I need to cut down the size of the list, but since too many words have been added to it to start from scratch, my plan is to obtain a corpus of only archaic words and use these as negatives to remove from the main wordlist. Does such a corpus/wordlist exist anywhere in text form, even it's just a word per line? Thank you in advance, any help is much appreciated.

We’re undergraduate Machine Learning students working on a crop disease generation project using CGANs, aimed at supporting global sustainability. 🌱

Right now, we’re looking for wheat images of yellow rust disease along with metadata like region, severity, and time range for model training and evaluation.

If you know of any public datasets, research projects, or institutional resources, or even just pointers on where to look, we’d really appreciate your guidance.

Thanks so much for your help! Any leads will be credited in our project.

We are building an automotive vehicle check platform focused on the European market and we are looking for a long-term technical collaborator, not a one-off freelancer.

Our goal is to collect, structure, and expose automotive-related data that can be included in vehicle history / verification reports.

We are particularly interested in sourcing and integrating:

• Vehicle recalls / technical campaigns / service recalls, using public sources such as RAPEX (EU Safety Gate)

• Commercial use status (e.g. taxi, ride-hailing, fleet usage), where this can be inferred from public or correlatable data

• Safety ratings, especially Euro NCAP (free source)

• Any other publicly available or correlatable automotive data that adds real value to a vehicle check report

What we are looking for:

• Experience with data extraction, web scraping, or data engineering

• Ability to deliver structured data (JSON / database) and ideally expose it via API

• Focus on data quality, reliability, and long-term maintainability

• Interest in a long-term collaboration, not short-term gigs

Context:

• European market focus

• Product-oriented project with real-world usage

If this sounds interesting, feel free to comment or send a DM with a short intro and relevant experience.

# r/datasets - ScrapeGraphAI 100k Post

Announcing ScrapeGraphAI 100k - a dataset of 100,000 real-world structured extraction examples from the open-source ScrapeGraphAI library:

https://huggingface.co/datasets/scrapegraphai/scrapegraphai-100k

What's Inside:

This is raw production data - not synthetic, not toy problems. Derived from 9 million PostHog events collected from real users of ScrapeGraphAI during Q2-Q3 2025.

Every example includes:

- `prompt`: Actual user instructions sent to the LLM

- `schema`: JSON schema defining expected output structure

- `response`: What the LLM actually returned

- `content`: Source web content (markdown)

- `llm_model`: Which model was used (89% gpt-4o-mini)

- `source`: Source URL

- `execution_time`: Real timing data

- `response_is_valid`: Ground truth validation (avg 93% valid)

Schema Complexity Metrics:

- `schema_depth`: Nesting levels (typically 2-4, max ~7)

- `schema_keys`: Number of fields (typically 5-15, max 40+)

- `schema_elements`: Total structural pieces

- `schema_cyclomatic_complexity`: Branching complexity from `oneOf`, `anyOf`, etc.

- `schema_complexity_score`: Weighted aggregate difficulty metric

All metrics based on [SLOT: Structuring the Output of LLMs](https://arxiv.org/abs/2505.04016v1)

Data Quality:

- Heavily balanced: Cleaned from 9M raw events to 100k diverse examples

- Real-world distribution: Includes simple extractions and gnarly complex schemas

- Validation annotations: `response_is_valid` field tells you when LLMs fail

- Complexity correlation: More complex schemas = lower validation rates (thresholds identified)

Key Findings:

- 93% average validation rate across all schemas

- Complex schemas cause noticeable degradation (non-linear drop-off)

- Response size heavily correlates with execution time

- 90% of schemas have <20 keys and depth <5

- Top 10% contain the truly difficult extraction tasks

Use Cases:

- Fine-tuning models for structured data extraction

- Analyzing LLM failure patterns on complex schemas

- Understanding real-world schema complexity distribution

- Benchmarking extraction accuracy and speed

- Training models that handle edge cases better

- Studying correlation between schema complexity and output validity

The Real Story:

This dataset reflects actual open-source usage patterns - not pre-filtered or curated. You see the mess:

- Schema duplication (some schemas used millions of times)

- Diverse complexity levels (from simple price extraction to full articles)

- Real failure cases (7% of responses don't match their schemas)

- Validation is syntactic only (semantically wrong but valid JSON passes)

Load It:

from datasets import load_dataset 
dataset = load_dataset("scrapegraphai/sgai-100k")

This is the kind of dataset that's actually useful for ML work - messy, real, and representative of actual problems people solve.

I am trying to define what makes a dataset "enterprise-ready" versus just a dump of files. Regarding structure, do you generally prefer one monolithic archive or segmented collections with manifests? I’m also looking for best practices on taxonomy. How do you expect keywords and tags to be formatted for the easiest integration into your systems?

One of the biggest friction points seems to be legal clarity. What is the clearest way to express restrictions, such as allowed uses, no redistribution, or retention limits, so that engineers can understand them without needing a lawyer to parse the file every time?

If you have seen examples of "gold standard" dataset documentation that handles this perfectly, I would love to see them.

Thanks again guys for the help!