How We Scaled Data Exports from 200K to 2M Rows Without Crashing Our Servers

At Chapa, one of the biggest engineering challenges we faced wasn’t building fancy payment flows or integrations — it was something much more "boring" but equally critical: data exports.

Our internal teams relied heavily on database exports for analysis, compliance, and reporting. Initially, this worked fine when exports were in the range of 50K–200K rows. But as the company grew and data piled up, the requirements quickly ballooned to millions of rows. That’s when the trouble began.

---

The Struggle

We were using PHP Laravel for this task, since much of our internal tooling was already built on top of it. But Laravel’s export handling started to break down when we pushed it beyond 200K rows:

• Scripts would timeout or crash the server
• Memory usage skyrocketed
• Exports took painfully long to complete

The obvious option was to switch the language for this task entirely — maybe move to Python or Go. But this was not a decision we could take lightly. Rewriting parts of the system in another language risked affecting the user experience, something we couldn’t compromise.

We needed another way.

---

The Breakthrough

Instead of replacing Laravel, we asked ourselves: what if we could combine the strengths of different languages?

That’s when the idea clicked.

We built a mini Python program dedicated to handling the heavy lifting of the export. Here’s how it works:

1. Python does the dirty work

   • Directly queries the database
   • Streams and writes the results efficiently to the filesystem

2. PHP stays in charge

   • The Python process is executed inside a Laravel script
   • Once Python finishes, it signals back to PHP

3. User flow remains seamless

   • Laravel takes over again
   • Sends an email notification with the exported file to the appropriate users

Everything runs within a single PHP script — no HTTP requests, no gRPC calls, no complicated orchestration. Just a neat handoff between PHP and Python.

---

The Results

The outcome was better than we expected:

• Export capacity increased from 200K → 2M+ rows
• Export speed improved by 3x
• No more server crashes
• A seamless experience for end-users

---

Lessons Learned

• Don’t be afraid to combine languages when needed
• Use the right tool for the right job (Python for heavy data handling)
• Always prioritize stability and user experience

---

This experience taught us that sometimes the best solution is not a full rewrite, but a smart integration of technologies.