Big Data Real Estate Search Engine

This project started with a painful, yet incredibly common problem: Search was just too slow, and it was only getting worse as our data grew.

We were building a real estate platform housing around 10k–30k listings. Users needed to search properties using complex filters (price, location, attributes), view those results dynamically on a map, and book a specific time to visit the property.

At first, both the search and booking functionalities were handled directly within our main database. It worked fine—until it absolutely didn't.

The Reality of Legacy Constraints

This was not a greenfield project where we could just pick the perfect, modern tech stack from day one.

• We had a legacy database that we couldn't just abandon.
• We had no dedicated search engine initially.
• We were operating on a tight timeline, meaning we couldn’t introduce too many new architectural components at once.

Because of this, most of my decisions were about incremental, pragmatic improvements rather than chasing perfect architecture.

---

Problem 1: Search Was Too Slow

Initially, we relied on relational database queries with multiple filters, relying heavily on JOIN operations and LIKE queries for search. We had no caching in place.

As traffic grew, queries started taking ~500ms–1s. Combining multiple filters made it exponentially worse, and map-based location queries were punishingly heavy.

The 3-Step Evolution

1. Query Optimization (The First Fix) Before throwing new technology at the problem, I optimized what we had. I reduced unnecessary joins, added missing indexes for common filters like price and location, and simplified the query conditions.

• The Result: Performance improved by about 30–40%, but it still wasn't enough to survive under heavy load.

2. Introducing Caching (Redis) Next, we introduced Redis to cache popular search queries and frequently viewed geographic areas. However, we immediately hit a new problem: the cache was serving outdated data. Slots that were already booked were still showing up as available. To fix this, I added strict cache invalidation on update events and reduced the Time-To-Live (TTL) for sensitive data.

• The Trade-off: We accepted slightly more cache misses in exchange for much better consistency.

3. Moving to Elasticsearch (The Turning Point) Eventually, we hit the database's ceiling. It simply couldn't handle flexible filtering combined with heavy location queries. We finally introduced Elasticsearch. I indexed the listing data into ES and moved all search queries away from the DB, keeping the relational database strictly as the source of truth.

• The Result: Search latency instantly dropped to ~50–100ms, resulting in a drastically smoother filtering and map experience.

---

Problem 2: The Booking Race Condition

In this system, "booking" meant scheduling a property visit. It didn't involve processing a payment, but it still required absolute data consistency.

The issue was that two users could look at the same property, select the exact same time slot, and click "Book" at nearly the exact same millisecond. Initially, this resulted in double bookings.

The Fix: Concurrency Control

I utilized database constraints and transaction checks. Now, when a booking request hits the server, the system re-checks the slot availability within a strict transaction. It only inserts the booking if the slot is still available. Only one request succeeds; the other gracefully fails.

Accepting Imperfect Search Consistency

Even after fixing the backend booking logic, Elasticsearch (being eventually consistent) might still occasionally show an outdated, available slot to a user.

My decision was not to try and make the search engine perfectly consistent. Instead, the system just validates the slot again at the final booking step. If a user clicks a slot that was just taken a second ago, the system simply returns a clear, polite message explaining the slot is no longer available. This single decision simplified the overall system architecture immensely.

---

Problem 3: Map Performance

Displaying listings on the map caused the frontend to render too many data points, resulting in incredibly slow rendering and heavy backend queries.

The Solution:

• We restricted queries to only fetch listings strictly inside the user's current viewport.
• We explicitly limited the maximum number of results returned per query.
• We implemented basic clustering and grouping at the backend level before sending data to the client.

The Result: Map interactions became highly usable, and the query load on the server dropped significantly.

---

Architecture (Final State)

By the end of this evolution, we had cleanly separated our concerns into three distinct flows:

• Search Flow: User ➔ API ➔ Elasticsearch ➔ Results
• Booking Flow: User ➔ API ➔ Database (Strict Transaction Check) ➔ Booking Confirmed
• Sync Flow: Database Updates ➔ Async Job ➔ Update Elasticsearch

Key Trade-offs We Made

| Decision | Why We Did It | The Trade-off | | :--- | :--- | :--- | | Introduced Elasticsearch late | To reduce complexity early on. | Endured temporary performance issues. | | Used cache aggressively | To improve response times. | Risked serving stale data initially. | | Eventual consistency for search | To keep the system lightning-fast. | Search might show slightly outdated results. | | DB as booking source of truth | To ensure 100% correctness. | Higher latency during the booking step than reading from a cache. |

---

Real Lessons: What Actually Broke

Systems are rarely perfect, and things definitely broke along the way:

• Our cache initially served wrong availability data, frustrating users.
• We dealt with live race conditions resulting in double bookings before the concurrency fix.
• The relational database became a massive bottleneck much faster than we originally anticipated.
• Elasticsearch sync lag occasionally caused temporary UI inconsistencies.

What I Learned

This project proved that you rarely start with the perfect architecture—you evolve into it. A relational database works beautifully... right up until the exact moment it doesn't. I learned that caching is incredibly powerful, but it is actively dangerous without a strict invalidation strategy.

Most importantly, I learned that search and booking cannot be treated the same way. For search, speed is the priority. For booking, correctness matters infinitely more than speed.

What I’d Improve Next

If I were to continue iterating on this platform, my next steps would be:

• Implementing a better, fully event-driven cache invalidation strategy rather than manual triggers.
• Building smarter ranking algorithms based on actual user behavior.
• Adding tighter monitoring for slow queries and Elasticsearch sync lag.
• Improving the frontend UX to handle failed booking conflicts even more gracefully.