Meta revised a clause in its upcoming commercial terms after wording about using crawled website content to train its products drew criticism. The word “train” was removed from the revised US-English clause, but questions remain about the scope of crawling and the meaning of “advertising and business products.”
A website can be public without its owner agreeing to every possible use of its content. That distinction sits at the heart of a dispute over Meta’s upcoming commercial terms: what may the company crawl, and what may it do with the material it retrieves?
On October 15, Nieman Journalism Lab reported that Meta’s upcoming commercial terms contained a clause allowing it to crawl websites associated with business accounts and use the content for a range of purposes, including training. After criticism, Meta revised the US-English text. On October 17, The Verge reported that the explicit reference to training had been removed. The revised wording ties the stated purposes to advertising and business products, as well as product integrity and security.
What triggered the controversy?
The clause appears in Meta’s Commercial Terms, which set conditions for business and commercial use of Meta products. It refers to automated software—such as bots, spiders and crawlers—accessing domains, pages or URLs associated with a business’s products or services, including content made available through Meta products.
The earlier US-English wording went further. It said Meta could use content from those domains and URLs to “provide, develop, improve, research, train” and maintain the integrity of its products and services. Critics argued that this could be read as a broad permission to collect website content and use it in AI training, potentially affecting publishers and other businesses that maintain a presence on Facebook or Instagram.
This was a dispute about what the wording could permit, not evidence that Meta had already scraped a particular publisher’s site for AI training. Legal experts quoted by Nieman Lab disagreed with Meta’s reassurance, arguing that “develop” and “business products” could still be interpreted broadly. Meta, by contrast, said the terms were intended for its advertising and business products and did not create new rights over website owners’ crawling preferences. TechCrunch also reported on the revision and Meta’s response.
What exactly changed in the terms?
| Issue | Earlier US-English wording | Revised US-English wording |
|---|---|---|
| Stated purpose | Included “train” and referred broadly to Meta products, services and features under the terms. | Refers to advertising and business products, plus maintaining the integrity and security of Meta products. |
| Crawling permission | Allowed automated software to access and periodically retrieve content from specified domains, pages or URLs. | The permission to use automated software and periodically retrieve updates remains in the clause. |
| What the change proves | The earlier wording explicitly included training as a listed purpose. | The revised US-English clause no longer explicitly lists training in that sentence. The edit alone does not establish every possible use of crawled content. |
Still disputed: whether the remaining phrase “develop … advertising and business products” could be interpreted broadly enough to cover some AI-related development, and how the terms interact with robots.txt and other legal rights.
Meta’s position: the company said the consolidated crawling terms apply only to advertising and business products, and that the wording was clarified after people misunderstood it. Meta also said its use of web crawlers has not changed and that the update does not give it new rights that override robots.txt preferences.
Why critics remain concerned: Nieman Lab quoted legal experts who said the revised phrases “develop” and “advertising and business products” may still be broad enough to cover AI-related product development. That is a legal interpretation, not a court decision. The key unresolved issue is how the wording would be applied in practice.
One more caveat: the reports describe the US-English version checked on October 17, 2026. Some other language editions were reported to still show older wording at that time. Terms can change and regional versions may not update together, so business users should check the live version that applies to their account and region. The updated terms were reported as scheduled to take effect on October 30, 2026.
Does this mean Meta can no longer train AI on website content?
No—not based on this clause alone. The reported change removes the explicit word “train” from the revised US-English commercial-terms clause. It does not prove that Meta has stopped every form of AI training involving web content. Nor does it settle what other policies, agreements or laws may permit. The accurate conclusion is narrower: this particular clause was revised after public criticism, while its remaining wording is still being debated.
Another useful distinction is between a crawler’s ability to fetch a page and the permission to use the fetched material for a particular purpose. These are related, but not identical, questions. A website may be technically accessible to a crawler while the permitted uses of its content remain disputed under contracts, copyright law, privacy law or other rules. The answer can depend on the facts and the jurisdiction.
What about robots.txt and Meta’s crawlers?
robots.txt is a text file at a website’s root that tells crawlers which paths the site operator asks them to access or avoid. It is a useful signal for compliant crawlers, but it is not a password, a technical lock or a substitute for access controls. The rules are usually applied to specific crawler names, called user agents, so blocking one Meta crawler does not automatically block every other Meta crawler.
This is an important detail: Meta’s crawler documentation lists separate automated agents for different jobs. It describes Meta-ExternalAgent as crawling for uses that can include training foundation AI models or improving products through indexing. It describes Meta-ExternalAds as crawling for advertising and other business-related products. That documentation describes crawler purposes; it does not, by itself, resolve how the Commercial Terms clause should be interpreted. Check Meta’s official web-crawler documentation for current instructions.
Meta says website owners can continue using robots.txt to express their preferences. Legal experts quoted by Nieman Lab point out that robots.txt is generally a voluntary protocol, not an access-control mechanism that automatically creates a legal right. The practical lesson is to use it as one layer of crawler management, while protecting private or restricted content with authentication and server-side controls. For a rights dispute, technical settings alone may not settle the legal question.
What should publishers and website owners do now?
There is no one setting that answers every legal and technical question. These steps can help you manage access and keep a record of your decision:
- Read the actual terms that apply to your account. Check the relevant Commercial Terms and any applicable advertising or business-product terms, including the effective date and the regional language version.
- Inventory your public domains. List the websites linked from your Facebook Page, Instagram profile, product catalogue, advertisements and other business tools. Confirm that each site has an owner and a clear publishing policy.
- Review robots.txt by crawler name. Check Meta’s current user-agent names and decide which you want to allow or disallow. Test the rules carefully. Different crawlers perform different jobs, so blocking one may not block another.
- Protect private information separately. Do not rely on robots.txt to protect confidential files, customer records or paywalled material. Use authentication and server-side access controls.
- Keep records of policy decisions. Save a dated copy of the terms you reviewed, the robots.txt rules you chose and any communications with platforms. If your content has significant commercial or licensing value, seek advice from a qualified lawyer in the relevant jurisdiction.
Why this matters beyond Meta
The dispute highlights a larger issue for the open web. Publishers depend on search, social platforms and referrals to reach readers, while AI companies need access to information to build and improve products. Those interests can overlap, but they are not automatically the same. A page being public does not mean every possible reuse is uncontroversial, and a crawler being allowed to fetch a page does not settle whether a particular downstream use is lawful.
Clear terms matter because businesses should be able to understand what they are agreeing to. Clear crawler documentation matters because website owners need to distinguish agents used for different purposes. And transparent communication matters when a clause changes after public criticism: people should be able to identify what changed, when it changed, and which version applies to them.
The evidence points to a change in wording, not a final resolution of the wider debate. Meta removed the explicit reference to training from the revised US-English clause and says its crawler practices have not changed. Critics argue that the remaining language is still broad. Until the terms are interpreted or tested in a specific dispute, readers should keep those positions separate and avoid treating either as a settled legal conclusion.
Frequently asked questions
When do the revised terms take effect?
Reports on the previewed Commercial Terms say the update is scheduled to take effect on October 30, 2026. Check the live terms for your account because the document may change.
Did Meta remove all website-crawling permissions?
No. The reported revision removes the explicit training wording from the US-English clause, but the clause still describes the use of automated software to access and periodically retrieve content for advertising and business products and for product integrity and security.
Does removing “train” prove Meta will not use any crawled content in AI systems?
No. The text change is evidence of a change to that clause, not proof of every use across all Meta products, policies, data sources or legal arrangements.
Can I block Meta crawlers with robots.txt?
Website owners can use robots.txt to communicate crawling preferences to compliant crawlers. Follow the current instructions in Meta’s official crawler documentation and remember that robots.txt is not a substitute for authentication or legal advice.
Should a small website remove its Facebook or Instagram business presence?
That is a business decision, not an automatic requirement. Review the current terms, assess the value of your platform presence and content, and choose your crawler settings deliberately. Seek legal advice if the rights at stake are commercially important.
Sources and further reading
- Nieman Journalism Lab — Legal experts warn Meta’s new terms still grant broad scraping permissions (October 17, 2026). Reports the wording change and explains why some legal experts remain concerned.
- The Verge — Meta removes language that may have allowed it to scrape news sites (October 17, 2026). Includes Meta’s statement that the edit does not change how it uses crawlers.
- TechCrunch — Meta removes language that raised scraping and AI training concerns (October 17, 2026).
- Meta Commercial Terms — consult the current version and effective date.
- Meta for Developers — Web Crawlers — crawler descriptions and robots.txt guidance.
Editorial note: This article explains publicly reported wording changes and attributed statements. It is not legal advice. Contract language and crawler behaviour may change; verify the current documents before making business or legal decisions.
.webp)
0 Comments