FactGrid:Setup: Difference between revisions

From FactGrid
Jump to navigation Jump to search
(→‎Packages: new section)
(→‎QuickStatements: update (background mode works now))
(One intermediate revision by the same user not shown)
Line 6: Line 6:
This list is very incomplete. I hope to add to it in the future if any further packages are installed, but many existing installed packages are not recorded here.
This list is very incomplete. I hope to add to it in the future if any further packages are installed, but many existing installed packages are not recorded here.


* php-intl for recent versions of QuickStatements.
* php-intl for Unicode support in QuickStatements


== MediaWiki ==
== MediaWiki ==
Line 28: Line 28:
which is owned by the <code>www-data</code> group and group-writable.
which is owned by the <code>www-data</code> group and group-writable.


The bot has not been set up yet, so “run in background” doesn’t currently work.
Batches which the user requests to run in the background,
instead of directly in the browser,
are saved to the <code>factgridquickstatements</code> database,
to which the <code>quickstatements</code> SQL user has access;
an ugly hack in the <code>openDbTool</code> function in <code>/srv/magnustools/public_html/php/ToolforgeCommon.php</code>
overrides the normal (very Toolforge-specific) database access code
to instead open that database using the password residing in the <code>/srv/quickstatements/db-password</code> file,
which is owned by the <code>www-data</code> group and group- but not world-readable.
QuickStatements has also been patched to format batch links in its edit summaries
using the <code>quickstatements:</code> link prefix,
instead of the usual <code>toollabs:quickstatements/</code>;
the <code>quickstatements:</code> interwiki prefix was installed with the following command
(via the <code>maintenance/sql.php</code> script):
 
INSERT INTO factgridinterwiki (iw_prefix, iw_url, iw_local, iw_trans) VALUES ('quickstatements', '/quickstatements/$1', 1, 0);
 
The bot which actually processes the batches runs as <code>quickstatements-bot.service</code>,
loading batches from the database and sending the appropriate edit requests to the API.
(When it has nothing to do, it sleeps in one-second intervals.)
 
Make sure to run <code>systemctl restart quickstatements-bot</code> whenever code changes to QuickStatements are made,
otherwise the bot will not pick them up.


== Reasonator ==
== Reasonator ==

Revision as of 16:01, 24 October 2018

This page describes the technical setup of the FactGrid website and services. FactGrid currently runs on a single virtual server, and all the file system paths mentioned here refer to that server.

Packages

This list is very incomplete. I hope to add to it in the future if any further packages are installed, but many existing installed packages are not recorded here.

  • php-intl for Unicode support in QuickStatements

MediaWiki

MediaWiki is installed under /var/www/w/. Apache serves /var/www/ as document root, with the standard MediaWiki short URL setup to rewrite /wiki/ into /w/index.php.

The job queue is processed by the mediawiki-jobqueue.service unit.

QuickStatements

The git repositories for quickstatements and its dependency magnustools are cloned under /srv/, and symlinks in /var/www/ point into their public_html/ subdirectories. There is an oauth.ini configuration file in /srv/quickstatements/ (for this consumer, with a request modeled after the original Wikidata consumer), and a config.json file in /src/quickstatements/public_html/ describes the URL layout of the FactGrid site and selects FactGrid as the site to use. Logs go to /srv/quickstatements/tool.log, which is owned by the www-data group and group-writable.

Batches which the user requests to run in the background, instead of directly in the browser, are saved to the factgridquickstatements database, to which the quickstatements SQL user has access; an ugly hack in the openDbTool function in /srv/magnustools/public_html/php/ToolforgeCommon.php overrides the normal (very Toolforge-specific) database access code to instead open that database using the password residing in the /srv/quickstatements/db-password file, which is owned by the www-data group and group- but not world-readable. QuickStatements has also been patched to format batch links in its edit summaries using the quickstatements: link prefix, instead of the usual toollabs:quickstatements/; the quickstatements: interwiki prefix was installed with the following command (via the maintenance/sql.php script):

INSERT INTO factgridinterwiki (iw_prefix, iw_url, iw_local, iw_trans) VALUES ('quickstatements', '/quickstatements/$1', 1, 0);

The bot which actually processes the batches runs as quickstatements-bot.service, loading batches from the database and sending the appropriate edit requests to the API. (When it has nothing to do, it sleeps in one-second intervals.)

Make sure to run systemctl restart quickstatements-bot whenever code changes to QuickStatements are made, otherwise the bot will not pick them up.

Reasonator

The git repository for reasonator is cloned under /srv, and a symlink in /var/www/ points into its public_html/v2/ subdirectory. config.json is copied from config.json.template with some property IDs replaced with their FactGrid equivalent, a few replaced with “TODO”, and most other property IDs completely removed because they don’t apply to FactGrid. There are also minor uncommitted changes in vue.js (avoid CORS errors) and main-page.html (replace example items), though hopefully those should become unnecessary in the future.

Query service

Upstream instructions:

The query service source is cloned in ~factgrid/wikidata-query-rdf/, built using ant as described in the “getting started” document, and unzipped into /srv/wdqs-0.3.1-SNAPSHOT/ (to which /srv/wdqs/ is a symlink). RWStore.properties is edited to adjust the location of the journal file, which we have in /var/lib/wdqs/factgrid.jnl. There is also a nearly-empty mwservices.conf file ({"services":{},"endpoints":[]}) to avoid a warning if that file is missing completely.

The query service itself runs as the blazegraph.service systemd unit (run systemctl cat blazegraph to see the configuration files). Its standard output and error go to the journal, and can be viewed by administrators with journalctl -u blazegraph (add -e for the latest messages).

The updater for the query service, which reads updates from the wiki’s recent changes and applies them to the query service, similarly runs as blazegraph-update.service.

The query service UI is cloned in ~factgrid/wikidata-query-rdf/gui/. It can be built using npm run build, and the resulting build/ directory is then copied into /var/www/, with a symlink /var/www/query pointing to the latest version. A few of the files in the repository have uncommitted changes specific to FactGrid; before updating the GUI, they have to be stashed away.

git stash save &&
git pull &&
git stash pop &&
npm run build &&
now=$(date -Iseconds) &&
cp -a build/ /var/www/query-"$now" &&
ln -sfT query-"$now" /var/www/query # atomically update symlink
# optional: remove the old /var/www/query-* directory