Synopsis: As more and more purchases take place online, companies are naturally required to share information about pricing and inventory on their websites. When this information is publicly available, it can be scraped by both aggregators and competing businesses. Today we examine both of these phenomena and explore how they result in lower prices for everyday consumers.

Introduction

The summer’s winding down here in Grassland, and vacations have now come and gone. A few of our beloved team members may still be traveling contently abroad, but summertime will soon be over, and the cold grip of reality will set back in. Back to work, degen — September is here.

Looking back on the past few weeks, we probably spent them much as you did. Going to the beach, relaxing with friends and family, and most importantly of all, reflecting on the role that web scraping can play in bringing down consumer prices. Did airfare seem cheaper than usual to you? Hotels a bit less expensive than you remembered? What’s going on?

Something might be afoot here. Let’s investigate the forces at work and see if — as usual — all roads might lead back to Grass…

Tech and Savings

For nearly two years now, inflation has been on everyone’s mind. Money printers go Brrr and prices go up — it’s a tale as old as time. While price inflation is a near constant in modern life, though, there are a few prices that have actually come down in recent years. Let’s look at one example: used cars.

Looking at the chart, we can see that the average price for a used car has fallen dramatically over the past decade. From 2010 to 2020 — before COVID supply constraints set in — that number fell from over $30,000 to just under $20,800. Now, if CPI was trending up that entire time, what could account for the drop in price?

The argument we’re going to make today is that technological innovation has the potential to reduce consumer prices, and the type of web scraping performed on Grass does exactly that. We’ll get to explaining that in a second here. In the case of used cars, however, the innovation is simple. The reason used car prices came down was the internet.

Before the internet was introduced, used cars were generally sold on sheisty used car lots, notorious for questionable products and unscrupulous sales tactics. A used car salesman’s job was to sell you a $20,000 car for $30,000.

When the internet was introduced, it gave consumers a wealth of information about the actual value of their cars, while simultaneously transitioning the market to online marketplaces like Craigslist. Now able to access counterparties on their own, consumers were freed from extractive middle men and prices came down.

There’s a lesson there. Sometimes, all it takes for a more efficient market is for a new technology to make siloed information more available.

So now that we have that as a point of reference for technology causing prices to fall, how does Grass fit in? Are there any ways that prices are coming down in the present tense?

Fast forward to today…

Airfare and Hotels: Cheaper When Scraped

At some point this summer, each of us took a customary vacation. When we got back and talked about it, it occurred to us that a number of the things we spent money on actually seem cheaper now than they used to be. The more we thought about it, the more we realized: the same way that the advent of the internet brought prices down for used cars, web scraping and big data are working to get consumer prices down even today. We realized that Grass doesn’t just earn you money — it also saves you money.

The biggest savings? Airfare and Hotels.

Looking at this chart, the average price of domestic airfare peaked at just under $600 in 2000 before steadily declining to less than $400 today. A decrease of over one third! What could cause such a decline? Sure, there’s been deregulation in the airline industry, but this occurred decades ago. What happened around 2000 that could have affected prices this way?

If you guessed web scraping, congratulations. 2000 was the year the first web API was invented, making it possible for the public to access bulk data straight from a company’s website. This was right at the peak of the dot-com bubble, and fare aggregators like Orbitz and Expedia became overnight sensations for scraping every airline’s website and delivering the cheapest option. For a lot of people, it was the first time the internet demonstrably made their lives better. A $600 flight now cost $550, all because previously siloed data could now be accessed and aggregated — thanks to the gift of web scraping.

Another place we’ve seen web scraping exert an influence is hotel prices.

For the past decade, hotels have faced a formidable opponent with the rise of rental apps like Airbnb. In the same way that used car prices came down when the early internet connected buyers and sellers, Airbnb emancipated consumers from having to pay hotel chains by connecting individuals with one another. You could make a strong argument that this simply enabled Airbnb to step in as an extractive middle man themselves, but the point remains.

Looking at the map above, though, hotels are now less expensive than Airbnbs in about half of the cities examined. So what gives?

The answer, once again, comes down to web scraping. Recently, we explored how companies use networks like Grass to scrape the web for their competitors’ prices, and that’s exactly what’s going on here. Airbnb may have originally succeeded because of its decentralized design, but this has ironically become a vulnerability: because its booking services exist entirely online, all of its prices are available to shrewd competitors! Hotel chains have recently started scraping Airbnb.com for pricing data and dropping their prices to compete — just as we described in the prior report. In five of the American cities on the map, hotels are now cheaper than Airbnbs!

There’s naturally a lot of noise in data like this — regulations, the cost of commercial vs. residential real estate, et cetera — but it does show one more example of web scraping as the primary weapon in today’s increasingly competitive wars of corporate intelligence.

Conclusion

These are only a few of the ways that web scraping has entered daily life, but we can identify one theme: whenever companies have access to greater information, it creates a more competitive environment and ultimately brings prices down for normal people. When it’s a vacation we’re talking about, this is great news for anyone. So remember that beyond the passive income you earn from running the web app, Grass can also affect our lives in bigger ways — even when you’re actually out touching grass.

Authored by Yield Aggregator (for Frogs Anon) & Wynd Network Team