Build Your First AI Website Agent in Under 5 Minutes

In this article, I'll show you how to build an AI agent in just five simple steps.

Have you ever landed on a website looking for a quick answer, only to find yourself endlessly scrolling? What if you could create an agent that automatically handles this for you? Simply provide a URL and a question, and it will read the entire page to deliver an answer in moments. Let's dive into the code.

Step 1: Setting Up the Environment

First, we need to install the necessary packages to create our agent. langchain-community is essential for using Langchain's built-in features like agents and tools, while duckduckgo-search enables real-time web searches.

pip install langchain-community duckduckgo-search beautifulsoup4

Next, let's import the required modules. We'll use os for managing environment variables securely, and requests and BeautifulSoup to fetch and parse text from web pages. For building the agent itself, we need initialize_agent, Tool, and AgentType from Langchain, along with ChatOpenAI to leverage OpenAI's language models.

import os
import requests
from bs4 import BeautifulSoup
from langchain.agents import initialize_agent, Tool, AgentType
from langchain_openai import ChatOpenAI

Note: Your API key should always be kept secure and never exposed publicly. We'll set up an environment variable to handle it safely.

os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY_HERE"

Step 2: Extracting Text from a URL

For this step, we'll define a function to extract clean text from a given URL. This function will take a URL as input, fetch the page content, and use BeautifulSoup to parse the HTML and strip away all tags, leaving only the raw text.

def extract_text_from_url(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')
    return soup.get_text()

Step 3: Creating a Powerful Q&A Tool

Now, let's build a powerful tool for our agent. This tool will accept a URL and a question, extract the relevant text from the webpage, and generate an answer.

To handle the combined input (URL and question), we'll create a function that first splits the input string using a separator like a comma. It then uses our previously defined function to get the website's content. With the content and the user's question, we'll construct a prompt for the LLM to generate a precise response.

def website_qa_tool_func(input_str):
    url, question = input_str.split(',', 1)
    content = extract_text_from_url(url.strip())

    prompt = f"Based on the following content, please answer the question.\n\nContent:\n{content}\n\nQuestion: {question.strip()}"

    llm = ChatOpenAI(temperature=0)
    response = llm.invoke(prompt)

    return response.content

Next, we'll wrap this function into a Langchain Tool. When creating a tool, three parameters are crucial: - name: A unique identifier for the agent to call the tool. - func: The function the tool will execute (in our case, website_qa_tool_func). - description: A clear explanation of what the tool does, guiding the agent on when to use it.

This description is vital for the agent to correctly identify the tool for the job.

website_qa_tool = Tool(
    name="Website Q&A Tool",
    func=website_qa_tool_func,
    description="Use this tool to ask questions about any website content. The input should be a comma-separated string with the URL first, followed by the question."
)

Step 4: Initializing the Agent

It's time to initialize our agent. We will equip it with the Q&A tool we just created and an LLM for its reasoning capabilities. We'll use a zero-shot-react-description agent, which can determine the sequence of actions based solely on the tool descriptions without prior training. We'll also set verbose=True to see the agent's thought process and handle_parsing_errors=True to make it more robust.

llm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo")
tools = [website_qa_tool]
agent = initialize_agent(
    tools,
    llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True,
    handle_parsing_errors=True
)

Step 5: Running the Agent

With our agent ready, let's see it in action. We'll ask it a question about a specific website. For this example, the agent receives the query and correctly identifies that it needs to use the Website Q&A Tool. It proceeds to execute the tool, analyze the content, and formulate a response.

query = "https://interviewkickstart.com, Is there any course to learn ML?"
agent.run(query)

The agent's output shows its thought process, culminating in a clear and accurate answer:

Yes, the website mentions a flagship machine learning course designed for engineers with over 5+ years of experience. The program features individualized teaching, technical coaching, mock interviews, career skills development, and problem-solving techniques.

And there you have it! We've successfully built a website Q&A agent in five straightforward steps:

1. Environment Setup: Installed and imported all necessary libraries.
2. Text Extraction: Created a function to scrape text from a URL.
3. Tool Creation: Built a custom tool for the agent to perform Q&A on a website.
4. Agent Initialization: Set up the agent with the tool and an LLM.
5. Execution: Ran the agent and received an intelligent answer.

This process simplifies how we can interact with web content, turning a tedious search into a quick query.