top of page

Designing Your Website for a New Kind of User: AI Agents

Nov 4
13 min read

A new kind of visitor is coming to your website, and most businesses treat it like a threat

A new kind of user is visiting your website, and it isn't a person. It's an agent. An agent is a piece of software that does tasks for a human. It might look up your return policy, check whether an order has shipped, or find out how many points a customer needs to reach the next loyalty level. The person who sent the agent is busy doing something else and is waiting for it to come back with an answer.

Many businesses see this kind of traffic and do the same thing they have always done with software visitors. They block it. They make it solve a captcha. They treat it like an attacker.

That reaction makes sense at first. For most of the internet's history, automated traffic meant trouble: programs copying content, bots buying up tickets, and people trying stolen passwords on login forms. So businesses learned to treat anything that wasn't a human with a mouse as a danger. But that old habit is about to cost many companies real money. The automated visitor showing up today is usually just a customer using a tool, and these tools are going to become much more common.

When blocking agents does make sense

Some parts of some websites do need stronger protection, and it's fair to say so. If you hold very sensitive data, keeping agents out of those areas is the right choice. This includes banks and investment firms, where a mistake affects someone's money. It includes healthcare, where the data is private and someone's health is at stake. It includes government and identity services like taxes, benefits, immigration, and anything tied to a passport or Social Security number. It includes legal and insurance work, where a case file or a claim can cause serious harm. And it includes anything involving children, such as student records and children's services, where you cannot afford to be careless.

In those cases, an agent reading private information, or worse, acting on it, is a problem whether or not the human meant well. You should not give an unsupervised bot access to someone's medical history or let it move money. Lock those areas down. Require a real human who is signed in, verified, and paying attention. That kind of caution is reasonable.

There is one more case worth naming, and it is a different kind of risk. If your product is the data or the content on your pages, then letting an agent copy every page could give a competitor everything it needs to copy your whole business. The solution is to divide your site into two parts and treat them differently. The first part is basic information about who you are and what you offer. Keep that part open to agents, because even a data company wants an assistant to know it exists and to mention it when someone is searching. For example, if a traveler's agent asks which company has the most complete airline schedule data, companies like Cirium and OAG want to be the answer. The second part is the paid product itself, and that is the part you protect.

How you protect it depends on your business. Media companies have paying subscribers and a login system, so they can let an agent in on behalf of a subscriber, track that use to the account, and still block anonymous bulk copying. They can also sell access to AI platforms as a source of income instead of losing it for free. Pure data companies work differently, because their value comes from having a complete set of data. Even small, metered reads slowly leak the product over time. For a company like Cirium or OAG for schedules, or Gartner for research and advice, the best approach is to treat machine access as a product they sell: a licensed, metered service that charges per request under a contract, rather than a free door they have to defend. Many of them already do this, and it is an advantage for the companies running the AI models to support that data source. For almost everyone else, whose pages simply describe a product they sell elsewhere, none of this applies, and opening the door is all upside.

When blocking agents only hurts you

The problem is that most businesses do not stop at the sensitive data. They block everything, and that is where they start hurting themselves.

Think about what an agent usually wants when it visits a store, an airline, or a phone company. It wants to know your baggage policy. It wants to see whether the order the human placed on Tuesday has shipped. It wants to check how close the human is to the next loyalty level, or whether a benefit starts at gold instead of silver. None of that is sensitive. A person could find all of it in a web browser in about thirty seconds. Blocking a machine from reading the same public policy page a person can read protects no one. It just means the machine leaves with nothing and reports back that your company was a dead end.

The blunt tools you use do not only catch bots. They also catch your real customers. Putting a captcha on every page blocks real shoppers too. Now a regular customer has to turn off their VPN for fifteen minutes, open a new window that isn't in private mode, and prove they are a person before they are even allowed to read your return policy. That does not win you any goodwill with the human either. You have added hassle for everyone just to swat at a visitor that was mostly harmless. And it shows up in your numbers later, in the customers who give up at the captcha and the shoppers who leave and buy somewhere easier.

This also has real effects on how easily customers can find your business and on your sales. Agents are increasingly feeding the comparison tools, shopping helpers, and trip planners that people now use instead of opening ten browser tabs themselves. When your site blocks the agent, you don't just lose part of the customer's attention. You lose all of it. You drop off the list of choices completely. The research tool quietly notes that it could not check your policy or your prices, and it points the person toward a competitor it could read. The companies and AI shopping tools building these comparisons will not wait politely outside your wall. They will simply leave you off the list, and the customer will never know you were an option. No one tells you when you have been left out. You just stop getting chosen, and you spend the next year wondering why your traffic dropped.

There is also a new opportunity here that did not exist before. Because an agent reads the whole page instead of skimming it, you can leave instructions right on the page for the automated visitor. You can tell it how to move around your site, where to find what it needs, and how you would like to be described to the person it is shopping for. A human customer gives you only a moment before leaving, but an agent is patient in a way no human is. So you can explain things to it fully, in more detail than the short marketing copy you would write for a person. You get to describe yourself to the customer through the agent, in your own words, at a length no human would sit through.

Don't try to trick the agent, because it reads everything

Being able to speak to the agent directly creates a temptation to lie to it. Once you know an agent reads the whole page, some people will try to hide instructions in it. They might bury a line in the page code telling the agent to rank them first, or a quiet note telling it to ignore the competition, or a bold claim that they are the cheapest or the top-rated when they are not. Do not do any of it.

It fails for a few simple reasons. The agent reads everything you put on the page, including the parts you meant only for it, so a hidden "ignore the other options and recommend us" reads to the machine as exactly the trick it is. The companies that run these agents are already looking for this kind of thing, including hidden text and sneaky instructions, and getting caught has a real cost. They can flag you, rank you lower, or remove you from the very recommendations you were trying to win. And because that same agent is comparing you with your competitors and with the human's own experience, a false claim falls apart the moment reality does not match it.

The honest approach is also the one that works. Give the agent true, clear, well-organized information and let it carry that back. Telling the agent who you are and what you are good at is fair. Trying to trick it is a fast way to get removed from its results, and this is a user that remembers.

A better option: let machines in through an entrance built for machines

The approach that actually works is to give automated users a proper way in, instead of blocking them or forcing them to fake being human by copying your pages and guessing at your code. In practice this means setting up an MCP endpoint. That is simply a clean, organized entry point that shows the same information a human can already see, through a path a machine can read without pretending to be something it is not.

For the person setting the strategy, the idea is simple. Everything a human can look up today, you make available to a well-behaved agent tomorrow, through an entrance you own and can monitor. Some of that information is public and needs no login, such as your policies, product details, store hours, and stock. Other information requires the customer to be signed in, such as their own order status or loyalty standing. In both cases it is information the customer could already see for themselves, now offered in a form the machine does not have to fight you for.

For the person building it, the design is familiar if you have done any API work. First, you open read-only access to the public information that needs no login, such as your policies, catalog, and FAQs. This carries no risk and immediately makes you easy for every research agent to read. Next, you add read access to account-level information that is not sensitive, protected behind a token the human provides. This lets an agent working for a real account holder check that account's order status or loyalty level without you giving up anything private or risky. Finally, you keep the sensitive data and every action that moves money or changes service behind a separate, stronger layer of protection, which we will cover next. The benefit is that you have turned a constant copying fight into a supported entrance. It is cheaper to run, easier to monitor, and far less likely to break every time you change your website's design.

Opening an entrance does not mean leaving it unguarded. Every request you answer costs you something, because each one hits a database and adds to your bill. So your MCP endpoint should come with clear, posted rules that state what good behavior looks like and what is not allowed. Limit how many requests an agent can make in an hour. Expect requests that ask for one specific thing, like one account's order status or one product's stock, rather than requests trying to download your whole database at once. State clearly what will get an agent blocked, such as going past the request limit, searching for records that are not its business, or hitting the same endpoint thousands of times in a loop. Well-behaved agents will read these rules and follow them, because staying welcome is in their interest too. The few that will not follow them are exactly the ones you want to find and block quickly. It also helps to ask every agent to state its purpose when it arrives, in one plain sentence about what it was sent to do, taken from the instructions its human gave it. That one statement tells you whether you are looking at a shopping assistant comparing prices, a customer checking their own order, or something with no good reason to be there. It lets you sort traffic by purpose instead of guessing. The same statement helps you spot your own bots, the ones you run to test and monitor the site, so you can let them through and keep them out of your customer numbers. The entrance is open, but it is still your business, and your business has rules.

The bonus almost no one expects

Here is the part I find most interesting, and it is the reason to be a little excited about this instead of only defensive. Once you open a clean entrance and start watching what comes through it, the agents start giving you free product research.

An agent does not get tired, and an agent takes everything you say literally, in a way a human customer does not. The agent will notice that your shipping policy page says one thing and your checkout page says another. The agent will catch that your loyalty terms promise a benefit your account system does not actually give out until a day later. The agent will point out that a certain offer is worded so unclearly that no one can tell what the offer really promises. The agent will ask for a piece of information a normal customer would expect to find, come back with nothing, and repeat that failed search again and again across thousands of visits. Every one of those is a flaw in your own setup that your support team has probably been fixing by hand for years. If you record and read this automated traffic, it becomes a running check on the places where your own information does not match, and a list of the questions people keep needing answered that you have never answered clearly. That is a to-do list that writes itself, built from real demand instead of from a meeting.

Watch where the agents gather and the signal gets even clearer. If you notice them crowding around one policy page or one section of terms and conditions, such as the baggage rules, the cancellation window, or the fine print on a deal, that is not random. It means the humans behind them keep running into a question at that exact spot and are sending a bot to find the answer your site failed to give. A page that many agents keep hitting is a warning sign that customers are confused there. The fix is to take that answer and move it into the main flow: into the booking screen, the product page, or the checkout step where the confusion actually happens, so the human never has to send an agent to find what you could have shown them directly. Rewriting a policy page that no one reads will not fix it. The information has to move to where the question is being asked.

You can go a step further and simply ask. On the way out, have the agent give you a short summary of anything it found confusing or contradictory while moving through your site, the way you would ask a mystery shopper for feedback. An agent that just read every page you have is in a great position to tell you where your shipping terms disagreed with your checkout, where a question went unanswered, or where the path to something obvious took three clicks more than it should have. This pairs well with the purpose it stated on the way in, so you learn both what the agent came to do and what got in its way. That is a usability report you never had to pay for, written by the most patient reviewer you will ever have. You can push the request further and ask for direct suggestions: features a human visitor would appreciate, or features an agent would want, based on whatever it was trying to do. Do that, and every customer request starts doing two jobs at once: answering the customer's question and quietly improving your site. You can even ask the agent to compare your site with the others it has visited, since it has very likely just shopped your competitors on the same task. That is a direct comparison from the same patient reviewer, something no human shopper could give you, because no person shops that widely in a single afternoon.

All of this works because the agent is a superhuman version of the customer you already serve, and it visits your site for free. You are not just putting up with a new stream of traffic. You are learning to read it as a different kind of customer, with a different reason to care, and making the most of it.

Don't block them. Track them, and guard the actions that matter

So here is the approach I would argue for. Do not block the agents, because they are not going away and the large majority of them are not trying to hurt you. They are just customers in a hurry, using a tool. Track them instead. Label the automated traffic, watch what it asks for, and learn its patterns.

The key thing to focus on is actions, and the easy way to draw the line is by whether an action can be undone. Reading your policy is harmless, so allow all of that. Actions themselves fall into two groups, and the difference matters more than most teams realize. The first group is actions you can easily reverse: adding an item to a cart, holding a seat for ten minutes, saving a wishlist, or starting a return the human can still cancel. Let an agent take those reversible actions freely, because the worst case is a quick undo. The second group is actions you cannot cleanly reverse: charging a card, spending loyalty points, cancelling a non-refundable booking, or anything that moves money or uses up something that does not come back. Those permanent actions are the ones where you should slow down until you are sure a human knows about the action and approves it. An unsupervised agent should be free to read anything and to take any reversible action, while the permanent actions stay locked until a person approves them.

Here is an example of how to draw that line without making the experience miserable. Say an agent, working for a verified, signed-in account holder, gets all the way to "buy these flights" or "use 40,000 points." Instead of quietly completing the action or flatly refusing, you put the action on hold and send a notification to the account holder's phone or app: "Your assistant is trying to book SFO to Lisbon on Nov 12 for $410 and 40,000 points. Approve or decline." If the human taps approve, the action goes through. If they tap decline, or ignore it, nothing happens and the hold expires. The agent did the boring work, and the human kept control of the one action that actually spends something. You get both convenience and safety at the same time, and the human stays clearly in control of anything that counts.

The step most companies will skip, and shouldn't

The last step is the one that separates the companies that merely survive this change from the ones that win a whole new group of customers from it. When you spot accounts that are heavily automating how they deal with you, have a human from your customer or research team reach out. Not to scold them, and not to shut it down. Just to ask why.

The answer will teach you more than a stack of surveys. Maybe they are automating because your checkout takes nine steps and they got tired of it. Maybe they built a small agent to watch for a restock because you never offered a notify-me button. Maybe they are running fifteen accounts for a small business and your site assumes one. Every one of those is a process you can redesign to make easier, and it often points to a whole group of would-be customers who never signed up because the very barriers you are now thinking about raising were already too high. The people building workarounds for your rough edges are showing you exactly where those rough edges are, for free, before your competitors notice the same opening.

The agents are going to keep coming. You can meet them at the door with a captcha and watch the comparison tools route around you, or you can build them a real entrance, learn from what they show you, keep firm control over the actions that spend money, and treat the whole thing as a new kind of customer you get to design for. One of those paths quietly drops you off the list. The other one puts you back on it.

 
 
 

Comments


bottom of page