homeresume
 
   

Automate finding apartment for rent or sale

December 28, 20212 min read

I have recently automated the process of finding apartments for rent or sale. Tech-stack I used for automation API includes Node.js, TypeScript, NestJS as a framework, Postgres as the main database, Redis as rate-limiting storage, and React with Next.js for the front-end. This blog post covers some of the obstacles throughout the journey.

Getting the data from providers

Some of the providers provide APIs, some of them have to be scrapped. Ads with incomplete info are skipped. There is one cron job that checks for new ads from every data provider, it stops when it gets already known ads from every data provider.

Showing only active ads

Some of the ads are validated when data is retrieved from automation API, during that process inactive ads are removed. Cursor-based pagination is used so there won't be any skipped ads in a case when some of the previous ads are removed due to their inactivity, also only a small portion of the ads are validated when the user wants to see the results for the chosen filters. Besides that, there is some offset for already validated ads, to not overload the data providers.

Ads for the same apartment from different providers

This is not solved, duplicate ads can be useful to check if there is a price difference between the ads and if the advertiser is the apartment owner, the agency as advertiser takes some commission when the apartment is rented or sold.

Demo

The demo can be checked out at https://flat-me.com

Željko Šević

Željko Šević

Software Engineer, Node.js Developer

 

© 2022