Automate finding an apartment for rent or sale

I have recently automated the process of finding apartments for rent or sale. Tech-stack I used for automation API includes Node.js, TypeScript, NestJS as a framework, Postgres as the primary database, Redis as rate-limiting storage, and React with Next.js as the front-end.

This blog post covers some of the obstacles throughout the journey.

Getting the data from providers

Some providers provide APIs, and some have to be scrapped. Ads with incomplete info are skipped. One cron job checks for new ads from every data provider, and it stops when it gets already-known ads from every data provider.

Showing only active ads

Some ads are validated when data is retrieved from automation API. During that process, inactive ads are removed. Cursor-based pagination is used so there won't skip any ads when some previous ads are removed due to their inactivity.

Only a tiny portion of the ads are validated when the user wants to see the results for the chosen filters. There is some offset for already validated ads to keep the data providers manageable.

Ads for the same apartment from different providers

This still needs to be solved. Duplicate ads can be helpful to check if there is a price difference between the ads and if the advertiser is the apartment owner or the agency.

Demo

Check out the demo here