eXtro Flatfile Generator - create static content

1. General Information

IMPORTANT: This component can only be used on live (production) sites, not on development sites!
Install this component as last component before golive of your site!
DO NOT install this component on development sites before golive, as this may lead to unexpected behaviour!

The eXtro Flatfile Generator can help save resources and make a website load faster, especially the so-called first byte can be dramatically improved.
This is done by crawling the site and creating static content flatfiles from the dynamic content, i.e. it creates real static content.

The theory behind this is simple:

If a (dynamic) CMS like Joomla is used for a website, then on every page view, the page content must be created newly *.
This of course uses a lot of computing, disk, database and network resources, since this has to be done on every page view.

In contrast, when using flat files, the webserver software (Apache, nginx, etc) only has to serve one file, i.e. static content, which has been pre-generated *.
This can be done much faster, since no PHP, database or computing must be done to serve the file.

For content that must be dynamic, like e.g. shop carts, exceptions can be defined.
Also, when a user logs in, the site switches back to dynamic content.

* Note: for every page of the site, this component creates a static flat HTML file within its SEF path as directory.
This means: if you have e.g. a valid page at your-domain.com/my-products/product1/productdescription.html , the component will create a directory /my-product/product1 , and within this directory a file called productdescription.html
This will be done for every valid URL found by the crawl process. Depending on the number of URLs found, this might create a lot of directories and static content files on the web storage.
This will be reflected in the backend main list view, each item in this list represents a static content file created in the webpackage, with its respective directory.
On deleting items in the list, the respective file will also be deleted. If you do not want certain files to be created, add them to the links exception list in the component options.
The webserver will serve the staic HTML file that gets queried by a frontend user much faster than when it would have to be generated by the CMS.

AGAIN: this component might create a lot of files and directories on your webspace (You can delete these items anytime by using the backend). If you do not want this, do not use this software.

2. System requirements / Technical information

The eXtro Flatfile Generator has these system requirements:

  • Joomla 3.5 or above
  • In the Joomla Global Configuration, these SEO Settings MUST be turned on: Search Engine Friendly URLs Use URL Rewriting Adds Suffix to URL
  • Cron jobs must be available on the server
  • The server must allow fopen wrapper connections to access the domain
  • The server must generally be properly configured
  • PHP execution time should be > 30 sec*
  • PHP memory should be > 128M*
  • PHP must be allowed to create directories and files in the web root directory and its subdirectories
  • In Joomla, you should switch off cache and do not use caching plugins. Also, do not use extensions that modify the .htaccess file
  • Your browser cache should also be disabled when configuring/crawling

When crawling the site, a webserver timeout message may appear, especially when using a slow hosting package - This is not a component error and is absolutely normal!
It merely states that the web package used cannot handle all pages at once. In such a case, the Flatfile Generator main list view should be reloaded - now a message regarding incomplete crawl should appear.
The crawling process can then be started again by clicking the Button Generate Static files.
These 2 steps should be repeated until this message disappers.

* PHP memory and execution time should anyway be larger than this value by default on a PHP version >= 5.5

3. Download and Installation

After purchase, you can download the the installation file by either clicking the download link in the confirmation email, or login to our site, go to your account, find your orders and there the download link in the respective order.

Next, unzip the installation file on your local computer, and install first the component in the Joomla backend, then the plugin, and activate the plugin.
Go to Extensions->Manage->Install , select the component zip file, and click Upload & Install, do the same with the plugin zip file.
Now go to Extensions->Plugins, find the Flatfile Generator System Plugin and activate it.

When both component and plugin are installed, and when the plugin is activated, the component is ready to be used.

4. Setting up the options for static content

When the component is installed, click Components->eXtro Flatfile Generator to load the component.
Now you should see the Flatfile Generator main view which lists your static content files.
Since this is the first time the component has been started, the list will be empty and you should see some notices regarding setup and crawling.



First, you should do the initial setup - for this, click the Button labelled Options in the upper right corner.

Now you should see the following Options:

  • Crawl password: This password must be set to prevent unauthorized users from starting a crawl / check process.
  • Combine CSS/JS files: CSS / JS files can be combined to reduce requests. This feature is considered experimental*.
  • Exception list CSS/JS: A comma separated list of patterns. If one of these patterns is found in the URL of a CSS/JS file, it will not be combined.
  • Link exception list: A comma separated list of patterns. If a URL to be crawled contains one of the patterns from this list, no flatfile will be creted from it.
  • Validity period: This is the maximum validity of generated files. After this period, the respective file will be re-generated.
  • Maximum files to check by cron: The check cron can be configured to limit the number of pages to check at once. This is the maximum number of pages to check per cron run.



When you've configured the Options, click The Button Save & Close to get back to the Flatfile Generator main list view.
Now, the notice regarding the password should no longer appear.

* This feature may be incompatible with other 3rd party extensions with similar functionality. In this case, leave combining of CSS/JS files off.

5. Crawling the site to create static content

After you've configured the component Options correctly, it is time to crawl the site.
The first (initial) crawl should always be done manually, i.e. by clicking the Button in the backend.



After clicking the Button Generate static files, a lightbox will be opened and the site will be crawled.
Wait until the crawling process has finished, then close the lightbox and reload the main list view.



Under some circumstances, the crawl process may display a webserver timeout message - this is not a component error!
This can happen especially when your site contains many articles and menu items, and your hosting package is not very fast.
In this case, simply reload the main list view and click the crawl button again, the crawling will continue where it stopped.

Note: the static content flat files generated by the component must be re-checked and re-crawled regularly (or else they would not reflect the content added/changed in the CMS) - this is why we added 2 different cron scripts.

6. Using the Crons

As already mentioned, the flatfiles must be checked and crawled regularly to reflect added articles/menu items and changes in articles.
If rechecking/recrawling is omitted, the site will still work, but it will always display the content from the first crawl (i.e. it might get outdated) - we do not recommend this!

2 types of Cron script have been added to the component:

The check cron:

This process checks whether existing flatfiles are still valid, i.e. if the dynamic content has changed.
The validity period can be set in the Options.
If the dynamic content has changed, or the validity period has expired, the file will be re-generated.
Additionally, the maximum number of files to check can be set in the Options, to prevent overloading the server - we recommend not checking all files at once on slow hosting packages.

The crawl cron:

This process crawls the site again to search for additional pages.
If additional pages are found, they will also be turned into flatfiles to improve site speed.

Note: you should at least use the check cron to reflect changes in the single pages.
If you know that there wont be many additional articles or menu items in your site, it may be enough to re-generate the existing static content and to manually use the crawl button in the backend.
However, if your site content changes regularly (e.g. blog, news, etc) you should use the cron for automatic static content file refresh also.

To use the check cron, use this URL:

//your-domain.com/index.php?option=com_static&view=staticcheck&tmpl=component&pass=**Your-Password**

replace your-domain.com with your actual domain name, and **Your-Password** with the actual password you set in the Options.
In your servers cron configuration, call this URL using e.g. wget or curl

To use the crawl cron, use this URL:

//your-domain.com/index.php?option=com_static&view=static&tmpl=component&pass=**Your-Password**

call it as described above using wget or curl

How often these 2 crons should run depends on several variables:

frequency of your articles changes
frequency of your menu changes
frequency new articles get added

The more often your content changes, the more often you should run the check cron.
The same applies for the crawl cron, with the menu/article add changes.

7. Updating the Joomla CMS when using the eXtro Flatfile Generator for static content

If your Joomla backend notifies you of a update of your Joomla core system, we strongly urge you to follow the following procedure to successfully update your site:

  • disable the crons (check and crawl cron) and make sure none is currently running
  • go to the Flatfile Generator backend, and delete all generated items
  • delete the Joomla cache, also purge the expired cache
  • update the Joomla core system
  • start a new (initial) crawl and create all existing pages new
  • enable the crons again
    • This procedure may also apply (in a similar way) when updating other 3rd party Joomla extensions e.g. components or modules