BugWP logo BugWP
Back to Community
Home / Community / General

How to Prevent or Fix WP-Optimize Modifying Your Robots.txt File

14 threads Sep 10, 2025

Content

Many users of the 'WP-Optimize – Cache, Compress images, Minify & Clean database to boost page speed & performance' plugin have reported issues where the plugin automatically adds rules to their site's robots.txt file. This can lead to duplicate entries, formatting errors, or conflicts with SEO plugins. This guide explains why it happens and provides the most effective solutions.

Why Does WP-Optimize Modify Robots.txt?

The plugin adds a specific rule to prevent search engines from indexing a file it creates: wpo-plugins-tables-list.json. This file, located in the /wp-content/uploads/ directory, contains an anonymous list of plugins and their associated database tables. The 'WP-Optimize – Cache, Compress images, Minify & Clean database to boost page speed & performance' team suggests this is a security measure to prevent exposing potential database structure information. It uses the standard WordPress robots_txt filter hook to add this rule, which is the correct method for modifying a virtual robots.txt file in WordPress.

Common Problems and Solutions

Problem 1: Duplicate 'User-agent: *' Lines or Malformed Format

If your SEO plugin or theme already creates a robots.txt file and adds a User-agent: * rule, WP-Optimize's addition can create a duplicate, leading to validation errors in tools like Google Search Console or Yandex Webmaster.

Problem 2: Cache Comments Appear in Robots.txt

In some older versions, a caching issue could cause HTML comments like <!-- WP Optimize page cache... --> to be prepended to the robots.txt output, malforming the file.

Problem 3: The File Appears to Be Missing

WP-Optimize does not create or delete the physical robots.txt file. It only filters the output. If your file disappears, it is likely unrelated to this plugin and may be caused by another plugin update, theme change, or file permission issue.

How to Disable the Robots.txt Modification

The most reliable method to prevent WP-Optimize from adding its rule is to use a code snippet in your theme's functions.php file. This solution has been confirmed by multiple users in the community.

add_action('after_setup_theme', 'wpo_remove_robots_txt_entry');
function wpo_remove_robots_txt_entry() {
    if (function_exists('WP_Optimize')) {
        remove_filter('robots_txt', array(WP_Optimize(), 'robots_txt'), 99, 1);
    }
}

Instructions:

  1. Access your WordPress site's files, either via FTP/SFTP or your hosting provider's file manager.
  2. Navigate to /wp-content/themes/your-theme-name/.
  3. Edit the functions.php file.
  4. Paste the code snippet at the very bottom of the file, before the closing ?> PHP tag (if one exists).
  5. Save the file.

Important: Use a child theme if you are not using a default theme (like Twenty Twenty-Four) to prevent your changes from being overwritten by a theme update. Always test code on a staging site first.

Alternative Method (Less Common)

Another filter was suggested in some threads, though its effectiveness was mixed. You can try it if the primary solution does not work for you.

add_filter( 'wpo_update_plugin_json', '__return_false' );

What If You See Cache Comments?

If your robots.txt file is showing caching-related HTML comments, the issue was addressed in a previous plugin update. Ensure your plugin is updated to the latest version. You can also try the following:

  • Purge all caches from the WP-Optimize dashboard.
  • Manually delete the physical robots.txt file from your server's root directory (if it exists) and let your SEO plugin regenerate it.
  • As a last resort, you can exclude the robots.txt file from caching. In the WP-Optimize cache settings, navigate to Advanced Settings and add /robots.txt to the "URLs to exclude from caching" textbox.

By understanding why the plugin modifies the robots.txt file and using the provided code solution, you can maintain full control over your file while continuing to use the performance features of WP-Optimize.

Related Support Threads Support