Web Scraping Reddit Search Results into a Bubble.io app
Web scraping Reddit search results into your Bubble.io app is now possible with Browse AI.
Scrape Reddit search results: Turn web data into app gold with Browse AI!
Master web scraping in Bubble: Create AI-powered robots and integrate with webhooks effortlessly.
Automate data extraction: Learn to trigger scraping tasks and process results using API Connector in Bubble.
Introduction to Web Scraping with Browse AI
If you've ever wanted to scrape search results from Reddit just like this, I'm going to show you how. I'm going to show you how we can use a web scraper called Browse AI. Here's the homepage. And they're a great web scraper because not only can you scrape individual pages and highlight the particular content you want to scrape from them, you can also monitor pages and you get updates through to your app. Now they've got a series of pre-built robots.
Creating an AI-Powered Scraping Robot
We're going to be using one of those in this demo. And I'm going to show you each step of how we can create an AI powered scraping robot in Browse AI, how we can run that from our Bubble site, and then how we can get the data back into our Bubble app using webhooks. All of that will be covered in this video. But before I launch into it, if you're learning Bubble and you want to watch more tutorials just like this one, then you can click the link down in the description to get access to our full tutorial library on our website, planetnocode.com.
Setting Up the Browse AI Robot
So let's go into our Browse AI account and we'll choose a pre-built robot. And I'm going to use Reddit, extract posts from search and say use this automation. Okay, search I'm going to use is called Bubble.io. This is just a demo. We want to get things working, check. It's all working. And because I'm going to keep my credit spend on the platform quite low as I'm going to be doing a series of demonstrations like this one, I'm going to say Max post five and then we start the extraction.
The Reliability of Web Scrapers
Now they're a good and they're bad web scrapers. You're basically doing battle with any defenses that the website owner has put in place to try and stop web scraping. And you'll find different platforms and web scrapers are more reliable than others. In my testing or Browse AI seems to be quite impressive. In fact, we get a response back very quickly.
Integrating with Bubble Using Webhooks
So how do we go about adding this data into our Bubble app? Well, we can go into integrate and we can set up a webhook. And a webhook is a way of saying when this task is finished in Browse AI, send a success message and all of the data through to Bubble. So I'm going to go into my Bubble app and I need to be in backend workflows.
Setting Up the Backend Workflow
This is a bit of a cluttered app. I use it for a lot of demos and I need to select backend workflows at the bottom. If you don't see that, you have to enable it in settings. But let's create a backend workflow because this is going to be a place that Browse AI can send data to. So I'll say web scrape, Reddit, and then I need to put it into what I like to call training mode.
Configuring the Webhook
So this is where the data is going to be sent. And notice in particular that at the end it's got initialize. This is when it is in training mode. We actually need to ensure that we go back and remove initialize when we actually want this to run automatically behind the scenes. So I've copied that into my clipboard and I'm going to paste in here and add in the webhook.
Running the Task and Receiving Data
So when task is finished, data is going to be sent to that webhook. I now just need to rerun the task, but I'm going to make sure that I'm in this detecting request data mode, otherwise it will be a waste and I'll just say run task. So it's going to show us once more what's going on. It's added to a queue and hopefully we'll get back a result just as quick as before.
Analyzing the Received Data
That one took slightly longer than the first demo, perhaps because of, like I say, doing battle with, in this case Reddit, who wants to protect the data on their site. And it seems like Browse AI had to retry a couple of times, but we've got the data. So hopefully that now means that if I go back into Bubble, we've got this update page and this is saying, here is all of the data and this is training Bubble, the structure of the data to get back.
Saving Data to the Bubble Database
And most importantly for us, we've got these, this list of posts. So we've got, yeah, here we go, alternative to Bubble and we've got all of this data about each post. So I'm going to click save because that's training my Bubble app what to expect back in the, back in the API data that sent through. So how do I save this to the database? Well, I'm going to just create a new one called web scrape once I make it private. No, doesn't matter. And then I'm just going to put in the bare basics. I'm just going to say title, which is text, and then URL, which is also text.
Creating a Workflow to Save Data
So now if I go back to my workflow, I can create a new thing, web scrape, add in all fields, and then here's the key thing, it gave me a list of data. And we've got other tutorials demonstrating how you can effectively create a loop in Bubble using another backend workflow. But for the sake of this, I'm just going to save the first result. So I look for captured list posts. I'm going to say first item title, first item post.
Testing the Data Flow
Okay, now let's run the test and see if we actually get data through into our Bubble database. But remember what I said before, which is that we have to update our webhook and remove, initialize. So we can't update, we can only delete. So I'm going to delete that, paste it back in, remove, initialize and add it in. And now let's try again.
Verifying the Data in Bubble
So we will run exactly the same. And what we're expecting to see in our Bubble database is that we get data created in that web scrape data type. So nothing's happened just yet. So let's go back here and it's queuing up and I'm going to skip ahead to when it's done, right, seems to be done. We've got our results back from Browse AI and hopefully this means that if we go into Bubble and perhaps we need to refresh this, we will see data in web scrapes.
Troubleshooting and Fixing Issues
Okay, we've got, we've got an issue, so let's go ahead and debug it. So I'm going to integrate. Oh, I know exactly why this is. Right, it's failed. Okay, I overlook something here, which is we have to basically say that this can be run without authentication. That's it. Otherwise you have to supply an API key, probably not secure in this case to just put an API key in another platform like Browse AI.
Retrying the Process
So let's just go ahead and run that whole thing again and hopefully this time it will be a bit more successful. So run a task. Okay, we got the results back. Hopefully this time we'll see them in our Bubble app. So once more I'm going to hit refresh and, and there we go. After a slightly slow load there from Bubble, we can see that we have successfully saved the title and URL into our Bubble app.
Conclusion and Next Steps
So there you go. That's how you can use Browse AI to scrape posts from Reddit based on the search results and save them into your Bubble app. Here's how you can trigger a AI web scraping robot that you've built in Browse AI and you can trigger it from your Bubble app. We need to be using the Browse AI API to do this and so I'm on my extract post from Reddit search, but it's going to be the same whichever one you use. You just have to swap in the robot id and so it gives us the data here to send across.
Using the Browse AI API
And I'm going to open up the API documentation and go into robots. And we want to tasks, yeah, task, run a robot. So this is what we need to send across to Bubble. So I'm going to copy, here we go, put it into the shell version. So we have to send a post request to this URL. So I'm going to copy that and go into my Bubble app. And we use the API connector to connect to services like this.
Setting Up the API Connector in Bubble
So go down here, add one in, call it Browse AI, and then we're making a post request to this URL, but we have to swap in the robot id. So let's make that dynamic robot id. So it says there how I can add in params into it, making it dynamic. I'm unticking private just so that if I was in the workflow, private is whether this data is accessible to the user. It doesn't really matter.
Configuring API Parameters
Think in this case if my users get access to my robot id, but that would allow me to dynamically swap in in a workflow if I wanted to. So let's paste in the robot id that we're working with and go back to the documentation. So I have to supply the robot id. I've done that. I have to authorize a bearer API key in the header. So in Bubble, the way I do that is I say private key and header authorization and then I would write in bearer and I'm going to paste in my API key, which I've got, I think it's this one here. Of course I'll be deleting this before publishing this video.
Adding API Action to Workflow
We'll call this one scrape. Content from Reddit needs to be an action so that I can run it in the workflow. And we have to include some parameters. So let's go back to the documentation and just see how we do that. So the two property names, does it give it an example here? Okay, right. So data input property, I'm going to copy all of that. Just so I have an idea of the layout.
Finalizing API Configuration
Paste it, that can probably add into parameters here. So the parameters we need to add in are here. We need search text and we need max posts. So let's try that, say search text. Ah, no, this isn't going to work because it's got to have, it's nested inside this key here, input parameters so I can't put it in there. So I've got what was it, search text and this is where I said Bubble.io and then there is another one I guess. Yes that would be a comma and that would be Maxposts.
Testing the API Connection
Now that's a number, but I think it probably wants me to send it as a string. Let's just try that. So I'm going to see if I get an error and I do that by running initialize call. No, I get a success code and so I'm going to click save and just so I'm really quick to show you what's going on here, if I go back and then into run tasks, we should see a new task is running. Yes, created by the API and it's running now.
Understanding Webhooks and Data Flow
So if I have my webhook set up, that data will come through to my webhook as soon as it's finished running. Because remember it can take a web scrape per 5 seconds or if it's got to change some settings spoof being a user, it can take a few minutes to get a result back. That's why we use webhooks, so that our Bubble app isn't waiting and when Browse AI has got the data, it's going to send it through to our Bubble app, through to that backend workflow using a webhook.
Get the Complete Bundle for Just $99
Access 3 courses, 390+ tutorials, and a vibrant community to support every step of your app-building journey.
Start building with total confidence
No more delays. With 30+ hours of expert content, you’ll have the insights needed to build effectively.
Find every solution in one place
No more searching across platforms for tutorials. Our bundle has everything you need, with 390+ videos covering every feature and technique.
Dive deep into every detail
Get beyond the basics with comprehensive, in-depth courses & no code tutorials that empower you to create a feature-rich, professional app.
Valued at $49
Valued at $89
Valued at $29
Valued at $99
Valued at $49
Can't find what you're looking for?
Search our 300+ Bubble tutorial videos. Start learning no code today!
Have questions?
We have answers!
Find answers to common questions about our membership plans, programs, and more.
We're here to help you launch your no code SaaS. Reach out to the team and we'll double check our vast library for useful content. We'll advise you on how we'd tackle the same problem and there's a good chance we'll record the video to help the wider community.
As a Planet No Code member, you'll receive a discount on our Bubble coaching sessions. Monthly members receive a 10% discount, while Annual members receive a 17.5% discount. To redeem your discount, simply log into your account and book a coaching session through our platform.
Our 8-week intensive mentorship program is designed to provide personalized guidance and support to help you accelerate your startup journey. You'll be matched with a startup expert who will work with you one-on-one to set goals, overcome challenges, and make rapid progress.
To apply for the Mastery Program, simply click the "Request Invitation" button on our pricing page and fill out the application form. Our team will review your application and schedule a call with you to discuss your goals and determine if the program is a good fit for your needs.
We accept all major credit cards, including Visa, Mastercard, American Express, and Discover.
While we don't offer a free trial, we do provide a 14-day money-back guarantee. If you're not completely satisfied with your membership within the first 14 days, simply contact our support team, and we'll issue a full refund.