Profile avatarPavel Svitek

How to create /llms.txt file in NextJS

What is llms.txt file

As artificial intelligence continues to reshape how we interact with digital content, optimizing websites to communicate effectively with AI systems is becoming increasingly critical. One innovative approach to enhancing this communication is through the use of an llms.txt file.

Much like the widely-adopted robots.txt, which guides traditional web crawlers, the llms.txt is a standardized Markdown file specifically designed to help Large Language Models (LLMs) quickly understand, index, and interact with your website's content. By clearly structuring and presenting your site’s information, llms.txt helps AI-driven systems more accurately and efficiently integrate your content into their responses and workflows.

Here's the official proposal — https://llmstxt.org/

Importance of llms.txt for SEO in the age of AI

In today's digital landscape, AI technologies and specifically Large Language Models are increasingly integrated into search engines, virtual assistants, customer support platforms, and countless other services. As users increasingly rely on these AI-driven tools to access information, ensuring your website is optimized for LLM interaction has become essential.

An llms.txt file not only enhances the visibility and accessibility of your website to these models but also improves the overall quality of user experiences driven by AI interactions. Adopting this practice helps position your website at the forefront of modern web development, ensuring it remains relevant and discoverable in an AI-centric future.

Which models use web search

Several AI models have integrated web search capabilities to provide users with up-to-date information.

  • OpenAI's ChatGPT: With the introduction of the Deep Research feature, ChatGPT can perform multi-step research tasks, retrieving and synthesizing information from the web to generate detailed reports.

  • Perplexity AI: This AI-powered search engine offers real-time web search integration, delivering answers with inline citations. Its Pro version provides access to advanced models like GPT-4 and Claude 3.7, enhancing its search and reasoning capabilities.

  • Google's Gemini 2.0: Google's latest AI model, Gemini 2.0, features autonomous agents capable of advanced searches and native image generation, aiming to solve multi-step problems efficiently.

  • You.com's YouChat: You.com has developed YouChat, a chatbot that integrates real-time web search results into its responses, providing users with up-to-date information and citations.

  • Mistral AI's Le Chat: Mistral AI's assistant, Le Chat, incorporates web search capabilities, enabling it to provide reliable and current information in its interactions.

These integrations demonstrate a trend toward combining AI language models with live web search functionalities, enhancing their ability to provide accurate and timely information.

2. Understanding the Structure of llms.txt

The llms.txt file is formatted in Markdown, a simple, readable text-based format widely used as input for Large Language Models. Markdown allows users to easily structure content using headings, lists, and links, making it straightforward for both humans and AI systems to parse and understand the information provided.

3. Integrating llms.txt into NextJS project

​Integrating the llms.txt file into a Next.js application is a straightforward process, applicable to both the App Router and Pages Router architectures. The key lies in utilizing the public directory, which Next.js employs to serve static assets. ​ nextjs.org

Steps to Integrate llms.txt:

  1. Locate or Create the public/ directory. Ensure that a public folder exists at the root of your Next.js project. If it doesn't, create one.​

  2. Place your llms.txt file directly into the public/ directory.​

  3. The file will be accessible via https://yourdomain.com/llms.txt.​

File/Path Structure

Here's how your project's structure should appear:

1
my-nextjs-project/

2
├── public/

3
│   └── llms.txt

4
├── pages/ or app/

5
│   ├── page.js (or index.js)

6
│   └── ...

7
└── ...

    Compatibility with App Router and Pages Router

    • App Router (app Directory): Introduced in Next.js 13, the App Router allows for a new file-system-based routing mechanism. Despite this structural change, the handling of the public directory remains consistent.

    • Pages Router (pages Directory): The traditional Pages Router also utilizes the public directory in the same manner.

    In both architectures, the public directory functions identically, ensuring that static assets like llms.txt are served from the root URL.

    Additional Considerations

    • Caching: Next.js does not apply aggressive caching to files in the public folder, as they may change over time. The default caching headers are set to Cache-Control: public, max-age=0.

    By following these steps, you can effectively integrate the llms.txt file into your Next.js application, regardless of whether you're using the App Router or Pages Router.

    Final thoughts

    Integrating an llms.txt file into your Next.js application is a proactive step toward enhancing your website's accessibility and relevance in an AI-driven digital landscape. By providing Large Language Models (LLMs) with a structured, Markdown-formatted overview of your site's content, you facilitate more accurate and efficient AI interactions.

    This not only improves the user experience but also positions your website to leverage the growing integration of AI technologies across various platforms. Implementing an llms.txt file is a straightforward process that can yield significant benefits, ensuring your content remains discoverable and effectively utilized as AI continues to evolve.