Web Scraping Google Search Results with Serper.dev
Web scraping Google search results into your Bubble app is easy with a service like Serper.dev. In the previous video I demonstrated how we can easily, quickly, less than five minutes, we add in the API to the Bubble API connector and we can then get results such as this. And in fact I'm really surprised it's dead quick we get back our lists and my search term query was something like oh best no code web app platforms and I'm getting back Zapier, I'm getting back g2.
Saving Search Results to Your Bubble Database
Okay in this Bubble tutorial video I'm going to show you how you can save this data to your Bubble app. I'm going to go through every step you need but before I launch into that Bubble tutorials if you're learning well there's no better place than planetnocode.com. We've got so many Bubble tutorials that they don't fit on our YouTube channel, a little bit cheeky to say but what I really mean is is that we have got more Bubble tutorial videos on our website planetnocode.com than you can find on our YouTube channel.
Creating a New Data Type for Search Results
So let's save these search results to our Bubble database. First thing I'm going to do is actually go back here and click manual because I just want to see the key link, the key data fields basically. So we'll just do title and link I think. So I'm going to go into data, I'm going to create new data type, I will say search results and add in those fields title of type text and link also of type text.
Setting Up the Page and User Interface
Okay I'm now going to add in a page now I'm not going to do any amazing design work here I should just say web scrape Google search results. I can't web scrape, web scrape Google will do and I'm now going to add in a button and an input with this search query and I'll say web scrape Google search.
Configuring the API Call
Okay now I get back a number of results and I can add in if I go into the playgrounds here I can say how many results I want to get back and it adds a parameter the default being 10 to the body or the raw data content of my API code. But yeah I know I'm going to get 10 back basically.
Creating a Backend Workflow
Now I could have one massive workflow which is saying save result one, save result two, save result three but I want to make it a bit of a better experience to my front-end users and also yeah I like if I'm going to iterate through a list of things I like to be able to make that number like n or x as in it's a variable it's a flexible number. So to do that I'm going to run a back-end workflow.
Setting Up the Backend Workflow
So I'm going to have one back-end workflow because I'm going to make this whole process run in the back-end it's not going to be public I will say web scrape Google search and I'm going to add in two parameters so add one parameter just say search query and then I add in my action so plugins SERPA.dev scrape Google.
Configuring the API Workflow
Now if you don't see that the number one reason you don't see it is because in plugins you have use as set to data and not action. If you swap it to action you might need to re-initialize the call but then you'll find it available as a workflow action. I'm going to swap this for my search query that's like my variable that I'm putting into this workflow and now I need another API workflow and I say save search result and I'm going to add in data create a new thing search result this is going to be run for every one of the ten search results or any number that I get back from SERPA.dev and so add in title add in link and then into here I need to also add in title and link and then link up the endpoint parameters title and link to say title and link.
Scheduling the API Workflow
So then on my web scrape Google I can add in here schedule API workflow on list one quick thing on back-end workflows you won't be able to access this or run this unless you're on a paid Bubble plan but we can run on a list and the list is Google scrape organic how do I know it's Google scrape organic but that's because if I go back here and initialize the call it's here organic list this is the list that I want to iterate through in order to save the results and then the list I want to run on is there and then I want to say save search results I want it just to run immediately so current date time if I leave the intervals box empty Bubble will self optimize if you're going to schedule a list of hundreds of things they're just all going to queue up and Bubble execute them as soon as it can and you can get a little bit of kind of stickage and pause there but that shouldn't really matter with these ten and then title is going to be when I get this option at the top this good scrape Google organic I'm now referring in terms to one of the results so I now say title and I say link obviously I get all of this extra data such as site links and position but you know you can add those in but I'm just keeping it simple for now.
Displaying the Results
And then I need a way of displaying it so I'm going to go back to my page Google web scrape and I'm just going to add in a repeating group and it's going to be off of search results and all I'm doing in order so this tutorial doesn't go on for ages is I'm just going to list all of the search results that I've created you might want to consider grouping them by having another data type called search and you would create that when this button is clicked and that way you can go through so if your user clicks the button three times you have three separate searches and each of those separate searches would be related to ten search results but for now just to prove that it's working I'm going to say do a search for search results not limit it so that's a zero ah doesn't work if I'm using fixed layout very nicely change that to 50 and then I'm going to add in some text search results title just add an align space that's nice and quick search results link I think that is now all going to work yeah add in the button workflow so what does this do this button triggers my API well my my back end workflow which is web scrape Google search not my individual safe because that is triggered by the web scrape Google search I'm going to say current date and time and then my search query is going to be the input on the page let's test that.
Testing the Web Scraping Functionality
So why don't I search for the not only get competitors coming up why don't we go for something like creative uses of chat GPT okay now you can see this is Bubble going through the 10 results and saving them one by one the reason I'm not being displayed fully I'll update that is because I've been lazy and I've used fixed layouts and fixed layouts cause headaches let's try that go back to my demo fresh the page okay there we go I've scraped all of those results for how you could use chat GPT creatively.
Conclusion and Potential Applications
So this is an incredibly powerful tool if your web app does anything in the realm of SEO and you want to create a web app even that allows your users to track their position you can do that this is the starting point for doing that by using the surper.dev API and we've covered it in basically 15 minutes how to add in Google search web scraping or you know accessing the Google search API into your Bubble application.